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I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

Web-based  and  distance  learning  techniques  are  increasingly  being  offered  as 
alternatives  to  conventional  instruction.  Since  students  now  have  more  opportunities  to 
complete  or  enhance  their  education  via  the  web,  techniques  to  attract  and  retain  this 
large  student  population  are  needed.  One  advantage  to  asynchronous  learning  is  that  it 
allows  class  schedules  to  be  flexible.  Students  are  able  to  view  class  lectures  and  other 
instructional  material  any  time  of  the  day;  whereas  with  conventional  instruction, 
students  must  be  at  the  specified  classroom  at  the  specified  time.  Another  advantage  of 
web-based  instruction  involves  the  simplicity  of  instructions  and  lectures  via  class  notes 
and/or  video  presentation.  These  materials  must  be  more  concise  and  succinct  due  to  the 
lack  of  face-to-face  communication  between  students  and  professors.  Conversely,  these 
advantages  can  be  disadvantages  if  proper  principles  are  not  followed  because  when 
students  are  self-paced,  they  may  lose  interest  and  not  complete  the  course.  This  is  also 
true  with  the  simplification  of  course  material.  If  not  done  correctly,  students  are  not  able 
to  extract  necessary  information  with  which  to  understand  complex  concepts. 


B.  MOTIVATION 

Techniques,  for  developing  high-quahty  web-based  instruction  are  developed 
from  three  disciplines:  (1)  instructional  design,  (2)  usability  engineering,  and  (3)  human- 
computer  interaction.  Instructional  design  principles  are  basic  guidelines  for  organizing 
and  presenting  class  material.  The  overall  goal  of  constructing  web-based  instruction 
guidelines  is  to  make  an  effective  learning  experience  for  students. 

Poorly  designed  computer  interfaces  may  impair  learning  or  impede  learning 
progress  and  efficiency.  Usability  engineering  impacts  the  quality  of  instruction  because 
if  students  cannot  use  the  system,  they  cannot  learn.  Likewise,  professors  cannot 
effectively  present  and  instruct  if  they  cannot  use  the  system.  The  professor  needs  to 
structure  information  into  the  system,  and  the  students  need  to  take  the  information  from 
the  system.  The  computer  cannot  stand  in  the  way  when  users  try  to  get  their  work 
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completed.  The  system  must  support  high  productivity.  This  goal  is  achieved  by 
designing  interfaces  that  are  consistent,  controllable,  and  predictable,  which  makes  it 
pleasant  and  effective  to  use  (Schneiderman,  1998).  In  all,  both  parties  should  be  able  to 
use  the  system  with  relative  ease. 

Human- computer  interaction  (HCI)  guides  the  design  principles  and  usability 
engineering  by  allowing  hdividuals  to  communicate  and  interact  with  each  other  by 
means  of  the  computer.  In  this  case  the  computer  is  not  only  the  “middle  man”  between 
the  professor  and  the  students,  but  also  between  students  and  students.  If  students,  for 
example,  spend  more  time  trying  to  learn  how  the  software  operates  rather  than  learning 
the  course  material,  they  lose  valuable  learning  time,  become  frustrated  with  the  system, 
and  lose  interest  in  the  instruction.  These  considerations  also  apply  to  educators. 

The  goal  of  this  thesis  is  to  develop  a  set  of  instructional  design  principles  for 
web-based  instruction  and  to  evaluate  those  principles  using  existing  evaluation  methods. 
This  thesis  will  then  devise  guidelines  from  the  instructional  design,  human- computer 
interface,  and  usability  engineering  literatures  and  evaluate  the  guidelines  with  a 
measuring  tool  to  evaluate  on-line  courses. 

It  is  not  the  intent  of  this  thesis  to  find  alternative  methods  of  long  distance 
education,  but  to  seek  out  ways  to  supplement,  complement,  and  enhance  web-based 
education  for  students  at  NFS. 


C.  RESEARCH  QUESTIONS 

This  research  sets  out  to  answer  several  key  questions.  First,  which  principles  of 
instructional  design  apply  to  web-based  instruction?  Since  web-based  instruction  is  a 
fairly  new  process,  there  needs  to  be  guidelines  for  instructors  to  use  when  putting  a 
course  online.  This  research  intends  to  introduce  a  method  of  derived  instructional 
guidelines  from  instructional  principles  for  web-based  instruction. 

This  leads  to  the  second  research  question:  Which  principles  of  general  web- site 
usability  apply  to  web-based  instruction?  In  other  words,  what  is  the  difference  between 
web-based  instruction  vice  general  web  interaction  that  makes  web-based  instruction 
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special?  Users  of  the  Internet  equate  web-based  instruction  and  general  web  interaction  as 
being  the  same  when  in  reality,  web-based  instruction  is  taking  a  formal  course  via  the 
web,  and  general  web  interaction  is  casual  browsing  on  the  Internet.  A  goal  of  this 
research  is  to  clarify  the  differences  of  usability  between  web-based  instruction  and 
general  web  browsing. 

Third,  which  principles  of  human- computer  interaction  apply  to  web-based 
instruction?  Computers  are  designed  for  various  users.  Certain  users  interact  differently 
with  the  computer.  Understanding  human- computer  interaction  should  improve  web- 
based  instruction  better  by  helping  designers  understand  how  the  average  student  learns 
via  the  web.  The  goal  of  this  research  is  to  clarify  HCI  principles  for  web-based 
instruction. 

Fourth,  what  are  the  typical  capabilities  and  constraints  associated  with  current 
web-based  instruction  software  frameworks?  When  software  developers  make  a  long¬ 
distance  education  product,  what  constraints  do  they  place  on  the  course  designer  that 
might  inhibit  web-based  instruction  rather  than  assist  students  to  learn?  This  research 
intends  to  identify  constraints  and  capabilities  of  software  products  and  how  they  affect 
web- based  instruction. 

Fifth,  what  are  valid  measures  of  web-based  instructional  quality?  Traditional 
instruction  has  been  around  longer  than  web-based  instruction.  Solid  concrete  guidelines 
have  been  established  to  measure  the  quality  of  instruction.  Regarding  web-based 
instruction,  we  do  not  know  how  to  measure  the  quality  of  content  for  web-based 
instruction.  A  goal  of  this  research  is  to  find  valid  methods  for  measuring  web-based 
instructional  quality. 

Sixth,  how  are  the  data  of  valid  measures  of  web-based  instruction  integrated  into 
models  to  enhance  the  quality  of  instructional  principles?  We  need  to  determine,  if  we  are 
measuring  navigational  design,  how  well  we  can  navigate  the  Internet  and  web-based 
site;  or  are  we  measuring  the  effectiveness  of  instructional  design?  Is  there  a  tradeoff 
between  navigational  design  and  instructional  design?  A  goal  of  this  research  is  to 
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determine  how  navigational  design  and  instructional  design  can  be  used  to  improve  the 
quality  of  web-based  instruction. 

D.  ORGANIZATION  OF  THESIS 

This  thesis  is  organized  into  the  following  chapters: 

Chapter  I:  Introduction.  This  chapter  includes  an  introduction  to  the  problem, 
motivation,  and  an  outline  for  this  thesis. 

Chapter  II:  Background.  This  chapter  contains  pertinent  information  regarding 
instructional  design,  human- computer  interaction,  and  usability  engineering. 

Chapter  III:  Theory.  This  chapter  focuses  on  the  theory  of  web-based  instruction 
and  how  instructional  design,  human  computer  interaction,  and  usability  engineering 
must  be  considered  when  developing  a  web-based  instruction  course. 

Chapter  IV:  Pilot  Evaluation  of  Principles:  This  chapter  summarizes  a  pilot 
project  that  initially  tested  the  principles. 

Chapter  V:  Method,  Results,  and  Analysis.  This  chapter  discusses  the  method  of 
testing,  the  results,  and  analysis  of  web-based  instruction. 

Chapter  VI:  Results  and  analysis.  This  chapter  discusses  the  results  from  the 

survey. 

Chapter  VII:  Conclusions  and  recommendations.  This  chapter  will  summarize  the 
final  results  and  thesis,  and  provide  potential  future  work  that  can  enhance  the  efficiency 
of  the  questionnaire 
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11.  BACKGROUND 


A.  INTRODUCTION 

In  order  to  understand  how  instructional  design  and  usability  principles  are 
intertwined  in  web-based  education,  it  is  important  to  understand  each  component 
separately.  This  chapter  begins  with  an  overview  of  human- computer  interaction, 
usability,  and  instructional  design,  a  discussion  on  why  the  areas  are  not  independent  of 
each  other  in  web-based  education,  and  what  has  been  accomplished  in  web-based 
education  with  regards  to  instructional  design  and  usability. 


B.  HUMAN-COMPUTER  INTERACTION 

Dix,  Abend,  Beale,  and  Finlay  (1998)  and  Preece  and  others  (1994)  define 
human- computer  interaction  (HCI)  as  a  “discipline  concerned  with  the  design,  evaluation, 
and  implementation  of  interactive  computing  systems  for  human  use.”  To  produce  good 
systems,  HCI  specialists  attempt  to  understand  how  people  effectively  operate  and  use 
computer  technology  in  their  environment  such  as,  the  work  place  and  home.  They  then 
develop  techniques  to  assist  designers  in  making  computer  systems  that  provide  good 
interaction  between  the  computer  and  humans  (individual  and  group).  The  main  emphasis 
of  HCI  research  and  design  is  to  not  only  place  the  individual  before  the  computer 
system,  but  also  keep  the  individual  in  mind  throughout  the  entire  system  design  process 
because  in  the  end,  the  user  has  to  use  the  product  to  accomplish  certain  tasks. 

1.  Disciplines  that  Contribute  to  HCI 

According  to  Preece  and  others  (1994),  11  disciplines  (Figure  1)  contribute  to 
HCI;  Computer  Science,  Cognitive  Psychology,  Social  and  Organization  Psychology, 
Ergonomics  and  Human  Factors,  Engineering,  Design,  Anthropology,  Sociology, 
Philosophy,  Linguistics,  and  Artificial  Intelligence.  Computer  Science  is  about 
optimizing  computer  efficiency;  Cognitive  Psychology  characterizes  processes  (e.g. 
perception,  attention,  learning,  thinking,  and  problem  solving)  in  terms  of  their 
capabilities  and  limitations;  Social  and  Organizational  Psychology  studies  the  nature  and 
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causes  of  human  behavior  in  a  social  context;  Ergonomics  (or  human  factors)  defines  and 
designs  tools  and  various  artifacts  for  different  work,  leisure  and  domestic  environments 
to  suit  the  capacities  and  capabilities  of  users;  Linguistics  applies  the  scientific  study  of 
language;  Artificial  Intelligence  (AI)  is  concerned  with  the  design  of  intelligent  computer 
programs,  which  simulate  different  aspects  of  intelligent  human  behavior;  Philosophy, 
Sociology,  and  Anthropology  are  used  to  design  and  evaluate  systems  in  order  to  provide 
a  more  accurate  descriptbn  of  the  interaction  between  users,  their  work,  the  technology 
they  use  in  their  environment;  Engineering  applies  the  science  to  produce  an  artifact; 
Design  assists  engineering  by  providing  creativity.  (Preece  et  al.  1994) 


Figure  1 :  11  Disciplines  of  HCI  From  Preece  et  al 


To  produce  a  system,  three  unique  areas  must  be  combined: 

•  The  1 1  disciplines  of  HCI, 

•  Tools  developed  by  researchers  and  consultants  based  on  those  disciplines, 
and 

•  participation  from  experts  themselves. 
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Figure  2  shows  how  a  design  eyele  normally  operates.  The  knowledge  and  skills  are  used 
to  develop  eomputer  tools.  The  design  team  uses  the  tools  to  assist  in  eonstrueting  a 
system  or  software  program.  After  the  system  is  eonstrueted,  an  evaluation  team  will  test 
the  system  and  its  usability.  (Preeee  et  al.  1994) 
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2.  Incorporating  HCI  into  the  Design  Process 

When  contending  with  HCI  design  issues,  Nielson  (1998)  and  Schneidermann 
(1998)  state  that  a  useful  design  philosophy  for  developing  user-oriented  human- 
computer  interfaces  is  to  consider  the  computer  as  a  tool  to  aid  the  user  in  accomplishing 
tasks.  A  tool  that  requires  more  time,  training,  and  effort  to  use  than  the  task  requires 
without  the  tool  is  a  poorly  designed  system.  Along  the  same  lines,  “Developers  of 
software  should  not  assume  that  they  know  what  the  users  do  because  they  are  also  users. 
This  type  of  designing  has  resulted  in  poor,  unusable  systems  that  wastes  hours  of  users’ 
time”  (Preece,  2000,  p.  112).  In  all,  designers  should  begin  to  think  like  users,  who  do  not 
care  about  how  the  computer  operates,  but  how  they  use  the  computer  to  achieve  specific 
goals. 

Incorporating  HCI  into  the  design  process  begins  with  support  from  management 
and  input  from  team  members.  The  teams  discuss  design  tradeoffs  that  can 
simultaneously  maximize  requirements  and  user  satisfaction.  This  is  the  foundation  for 
the  rest  of  project  and  is  very  critical  to  its  success.  Users  of  the  system  should  be  directly 
involved  in  the  initial  design  process  and  during  each  testing  phase.  Knowing  the  user  is 
also  critical  to  successful  system  design.  Developers  should  consult  experts  and 
researchers  to  understand  human  characteristics.  After  all,  “users  come  in  all  shapes  and 
sizes,  with  different  personalities,  abilities,  experiences,  resources,  and  needs”  (Preece, 
2000,  p.  124).  A  team  of  observers  should  visit  the  users’  work  environment  to  observe 
and  record  the  users’  daily  activities  and  how  they  interact  within  their  surroundings.  This 
helps  developers  to  understand  the  tasks  that  users  will  perform  with  the  system  in  their 
environment.  User  interface  guidelines  should  then  be  developed,  documented,  revised, 
and  maintained  for  each  project.  This  helps  future  projects  because  a  different  project 
team  is  able  to  research  and  understand  previous  work.  When  modifying  a  system, 
developers  also  have  a  base  to  build  upon  by  understanding  how  to  design  within  the 
varying  constraints.  System  and  software  developers,  analysts,  management,  and  other 
developers  should  be  trained  often  in  human-computer  interface  design  in  order  to 
increase  their  proficiency  in  their  area  of  expertise.  Training  reinforces  understanding  and 
use  of  interface  software  tools  that  enhance  the  consistency  of  the  interface  and  provide 
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an  environment  where  interactive  design  is  simple.  It  also  supports  program  modularity, 
software  libraries,  development  time,  and  cost.  These  aspects  are  critical  to  big  projects 
because  they  can  reduce  risk  while  adhering  to  the  development  cycle  schedule.  Testing 
and  prototyping  early  in  the  development  cycle  reveals  flaws  in  interface  requirements 
and  user  assumptions  on  how  they  will  use  the  system.  Testing  design  features  should  be 
performed  on  the  intended  user  population.  When  changing  and  refining  the  design  to 
users’  needs,  developers  should  be  flexible  and  patient.  Early  testing  of  the  design 
maintains  costs  and  helps  the  project  to  stay  on  schedule.  Problems  discovered  during 
testing  should  be  revised,  corrected,  then  retested,  and  documented  before  continuing 
onto  the  next  phase,  milestone,  and  project  completion.  (Brown,  1991;  Dix  et  al.l998; 
Hix  &  Hartson,  1993;  Preece  et  al.  1994) 

A  usability  study  determines  how  well  designers,  engineers,  and  developers 
incorporate  HCI  into  their  development  of  a  system.  This  is  where  HCI  envelops 
usability.  HCI  existed  before  the  term  “usability”  came  into  use.  HCI  principles  are 
typically  based  upon  a  user-centered  design  process,  and  certain  design  (human 
engineering)  guidelines  related  to  screen  layout,  use  of  fonts,  color,  highlighting,  etc. 
Also,  HCI  concerns  itself  with  the  use  of  input  devices,  display  devices,  and  their 
efficacy.  Usability  engineering,  on  the  other  hand,  assisted  in  bringing  the  idea  of  a 
design  metaphor  into  focus  and  placed  greater  emphasis  upon  building  uncomplicated 
menu  designs,  meaningful  control  labels,  and  improving  navigation  to  make  systems 
easier  to  understand  and  use.  Whereas  usability  deals  with  the  process  of  building  a 
system,  HCI  principles  cover  the  entire  design  process  of  building  a  system. 


C.  USABILITY 

Usability  is  a  key  issue  in  human- computer  interaction;  it  is  the  principle 
commonly  accepted  to  indicate  the  quality  of  a  user  interface  (Paralangeli,  Marchigiani, 
&  Bagnar,  1999).  Measuring  the  quality  of  the  user  experience  when  interacting  with  a 
system,  such  as  a  web  site,  software  application,  or  any  user-operated  device  is  an 
example  of  usability.  Nielson  (1998)  breaks  usability  into  five  characteristics;  (1)  Ease  of 
learning;  How  fast  can  a  new  user  sufficiently  learn  the  program?  (2)  Efficiency  of  use; 
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Once  the  user  learns  the  system,  how  fast  can  the  user  complete  tasks?  (3)  Memorability, 
How  effective  can  a  previous  user  accomplish  tasks  with  out  relearning  the  system?  (4) 
Error  frequency  and  severity.  How  many  errors  occurred  and  how  were  they  recovered? 
(5)  Subjective  satisfaction-.  Is  the  user’s  experience  a  positive  one?  All  systems  have  all 
five  characteristics  of  usability  and  all  need  to  be  considered  in  any  design  project. 


1.  Elements  Supporting  Usability 

There  are  three  distinct  elements  that  support  the  usability  characteristics.  The 
first  element  is  learnability,  which  is  the  ease  with  which  new  users  can  begin  effective 
interaction  and  achieve  maximal  performance.  It  is  measured  by  predictability,  which 
measures  past  performance  to  improve  future  interaction.  Synthesizability  evaluates  the 
effect  of  past  operations  on  the  current  state.  Familiarity  pertains  to  how  users  apply  their 
current  knowledge  and  experience  to  new  tasks  for  effective  interaction.  Sometimes 
“guessability”  is  when  users  learn  something  new  by  guessing  on  how  to  use  it.  If  the 
user  is  familiar  with  a  product  and  decides  to  use  a  similar  product,  the  user  is  more  likely 
to  have  a  higher  expectation,  and  is  able  to  “guess”  how  the  new  product  operates. 
Generalizability  relates  to  how  users  extend  their  knowledge  of  specific  behavior  to  new 
similar  situations.  Above  all,  consistency  pertains  to  likeness  in  behavior  arising  from 
similar  situations.  This  is  usually  accomplished  through  continuous  tasks  and  keeping  the 
tasks  simple  and  constant.  (Dix  et  al.  1998) 

The  second  element  supporting  usability  is  flexibility.  Flexibility  refers  to  the 
numerous  ways  users  and  the  system  exchange  information.  Dialog  initiative  allows  users 
to  communicate  with  the  system  within  a  constrained  environment  imposed  by  the 
system.  An  example  is  formatting  text  where  users  tell  the  system  what  font  and  style 
they  want.  Multithreading  is  another  form  of  flexibility  where  the  system  supports 
numerous  tasks  at  a  time.  Automatic  teller  machines  (ATMs)  effectively  permit  this  type 
of  behavior  of  user- system  interaction.  Task  migratability  relates  to  the  transfer  of  control 
between  system  and  users.  The  spell  checker,  for  instance,  displays  a  word  for  correction. 
After  users  decide  to  correct  a  word,  they  transfer  control  to  the  system  for  correction. 
Substitutivity  refers  to  how  the  system  can  substitute  values  upon  users’  requests  such  as, 

10 


changing  margins  or  numeric  values.  Finally,  customizability  refers  to  how  the  users 
modify  their  system  interfaces  to  their  liking.  Experienced  users  may  want  more  features 
whereas  novice  users  accept  the  standard  software  packages  offered.  (Dix  et  al.  1998) 

The  final  element  supporting  usability  is  robustness.  Robustness  is  how  the 
system  supports  users  to  achieve  a  set  of  goals.  Observability  affects  robustness  by 
allowing  the  user  to  evaluate  how  the  internal  system  operates.  Some  examples  are  how 
well  can  users  navigate,  how  well  do  the  defaults  perform  for  the  users,  and  how  well  the 
system  endures  through  constant  use.  Recoverability  assists  users  in  correcting  errors 
where  responsiveness  is  how  fast  the  system  can  respond  to  users’  intentions.  Finally, 
task  conformance  is  how  well  the  system  supported  the  user’s  goals.  (Dix  et  al.  1998) 


2.  Usability  Standards 

Standards  are  usually  set  by  national  or  international  organizations  to  ensure 
compliance  with  a  set  of  design  rules  by  a  large  community.  The  Department  of  Defense 
(DoD)  requires  that  software  developers  be  certified  according  to  the  Software 
Engineering  Institute  (SEI)  Capability  Maturity  Model  (CMM).  The  minimal  acceptance 
for  a  contractor  is  level  three  (defined  process).  At  this  level,  the  organization  has  a 
defined  process  definition  and  focus  of  their  software  development  program.  It  includes  a 
documented  and  implemented  training  program,  integrated  software  management, 
software  product  engineering,  and  peer  reviews.  Generally,  the  defined  process  is 
supported  by  upper  management  and  throughout  the  organization. 

The  International  Organization  of  Standardization  (ISO)  standard  9241  pertains  to 
usability  specifications  for  hardware  and  software  development  design.  It  gives  three 
specific  requirements  for  usability.  The  first  part  is  the  effectiveness,  which  is  the 
accuracy  and  completeness  with  which  users  achieve  certain  goals.  Examples  of 
effectiveness  include  quality  of  solution  and  error  rates.  The  second  part  is  efficiency, 
which  is  how  expended  resources  relate  to  the  accuracy  and  completeness  of  achieved 
goals.  Task  completion  time  and  learning  time  are  measures  of  efficiency.  Easily, 
satisfaction  is  how  comfortable  users  feel  when  interacting  with  the  system.  Was  the 

experience  positive  or  negative?  (Dix  et  al.  1998;  Schneidermann,  1998) 
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3.  Measuring  Usability 

Usability  testing  ensures  that  all  contractual  requirements  have  been  met,  helps 
maximize  the  usability  of  the  system  by  providing  feedback  during  development,  and 
provides  evidence  of  testing  in  cases  where  legal  issues  may  arise.  “The  greatest  benefit 
about  usability  testing  is  that  project  teams  can  identify  and  correct  errors,  which  can  help 
to  speed  up  delivery  dates  of  projects  while  reducing  project  costs”  (Schneidermann, 
1998,  p.  128).  Varying  degrees  of  system  errors  are  tolerated  during  testing.  However,  as 
the  number  of  system  inputs  increase,  the  testing  becomes  more  difficult.  Varying 
degrees  of  system  errors  should  be  capped  to  ensure  maximum  usability  of  the  system. 
Since  lines  of  code  are  increasing  within  software  programs,  it  is  important  that  testing  be 
increased  and  is  more  stringent  in  order  to  find,  correct,  and  reduce  the  number  or  errors 
and  bugs  within  the  software  development  cycle.  (Schneiderman,  1998) 

System  design  and  tests  should  include  expert  reviewers  who  provide  a 
comprehensive  report  on  system  problems  and  recommendations.  Models  of  expert 
reviews  are  heuristic  evaluations'.  Expert  reviewers  critique  an  interface  to  determine 
conformance  to  a  list  of  established  rules.  Guideline  reviews'.  Experts  review  the  interface 
in  accordance  with  the  organizational  guidelines.  In  consistency  inspections,  experts 
verify  consistency  across  a  family  of  interfaces,  checking  for  consistency  of  terminology, 
color,  layout,  input  and  output  formats.  They  also  ensure  that  training  materials  and 
online  help  are  also  comprehensible  for  users  when  they  use  these  services.  Cognitive 
walkthroughs  take  place  at  each  project  completion  phase  and  when  the  entire  project 
terminates.  After  ensuring  that  requirements  for  a  phase  have  been  met,  experts  simulate 
users  performing  benchmarking  tasks  that  physically  test  the  requirements.  This  ensures 
that  the  interface  is  capable  of  carrying  out  common  tasks.  In  a  formal  usability 
inspection,  experts  hold  panel  meetings.  The  individual  responsible  for  the  development 
of  code  discusses  the  code  before  a  moderator  and  a  panel  of  reviewers.  Then  the  panel 
reviews  the  presentation  and  discusses  the  interface’s  merits  and  weaknesses.  These  types 
of  formal  reviews  are  scheduled  at  project  milestones,  when  experts  are  available,  and 
when  the  project  team  is  ready.  These  reviews  are  for  critiquing  the  product  not  the 
individuals  who  have  developed  the  code.  The  reviews  help  the  development  effort  by 
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finding  errors  early  during  development.  This  ensures  that  at  product  completion,  the 
product  has  no  or  very  few  errors,  and  is  usable  for  the  customer.  If  time  does  not  allow 
for  formal  reviews,  quick  informal  demonstrations  of  the  product  provide  useful 
feedback.  (Dix  etal.  1998;  Preece  et  al.  1994;  Schneiderman,  1998) 

A  usability  study  follows  formal  testing.  During  this  phase,  surveyors  test  a  small 
portion  of  end  users  to  determine  how  effective  their  products  are.  The  end  users  are 
people  who  interact  with  the  interface,  and  they  determine  how  well  the  product  could 
serve  them  in  accomplishing  their  tasks.  Some  methods  of  a  usability  study  are  user 
ratings  where  the  user  rates  a  product  based  on  a  numeric  value.  Surveys  also  help 
developers  test  their  products.  Not  only  do  users  rate  the  product,  but  they  also  have  an 
opportunity  to  write  comments  after  each  rating.  A  survey  is  more  in  depth  than  a  rating 
because  users  explain  why  they  gave  that  particular  rating.  Another  method  to  conduct  a 
usability  study  is  through  an  interview.  The  interviewers  can  ask  certain  questions  and 
write  down  responses  to  rate  the  usability  of  the  product,  and  ask  additional  questions 
that  might  arise  during  an  interview  as  well  as  observe  and  write  down  comments  on  how 
the  user  interacted  with  the  system.  In  these  sessions,  the  surveyors  are  observing  the 
participants’  body  movements  and  comments  when  they  are  initiating  set  of  tasks. 
(Preece  et  al.  1994) 

D.  INSTRUCTIONAL  DESIGN 

This  section  is  mainly  based  upon  Charles  Reigeluth’s  book,  Instructional-Design 
Theories  and  Models:  An  Overview  of  their  Current  Status 

In  his  presidential  address  to  the  American  Psychological  Association  in 
1899,  John  Dewey  (1900)  called  for  the  development  of  a  “linking 
science”  between  learning  theory  and  educational  practiced  (Reigeluth, 

1983,  p.  5). 

Like  a  jigsaw  puzzle,  instructional  science  emphasizes  how  to  put  learning  models 
and  theories  together.  Instructional  design  (ID)  links  learning  theory  and  educational 
practice.  It  is  a  body  of  knowledge  that  prescribes  actions  to  optimize  desired 
instructional  outcomes  such  as  achievement  and  affect  (Reigeluth,  1983).  Reiser  (2001) 
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states  that  the  field  of  ID  and  technology  improves  learning  and  performance  by 
encompassing  the  analysis  of  learning  and  performance  problems,  design,  development 
implementation,  evaluation,  and  management  of  instructional  and  non- instructional 
processes.  By  understanding  instruction  and  the  five  processes  of  instruction,  one  is  able 
to  understand  how  ID  impacts  instruction. 


1.  Five  Processes  of  Instruction 

First,  instructional  theory  should  not  be  mistaken  for  learning  theory,  which  is 
concerned  with  what  happens  to  the  learner.  Instructional  theory  is  concerned  with  what 
the  teacher  does.  For  this  reason,  instruction  is  concerned  with  “how”  to  teach.  Design 
falls  under  instruction  because  design  relates  to  “what  method  will  the  instructor  use  to 
teach”  (Reigeluth,  1983).  Figure  3  shows  how  design  relates  to  instruction  in  regards  to 
the  education  process. 


Figure  3 :  Design  Relationship  Within  the  Education  Process  From  Reigeluth 

ID  involves  understanding,  improving,  and  applying  methods  of  instruction. 
Educators  are  concerned  with  what  methods  of  instruction  are  best  for  bringing  changes 
in  student  knowledge  and  skills  for  a  specific  core  content  and  student  population.  ID  is  a 
“blue  print”  for  what  method  should  be  used  for  the  course  and  students.  Instructional 
Development  takes  the  blue  print  and  begins  building  a  course  upon  the  design  through 
notes,  lectures,  and/or  lesson  plans.  Instructional  Implementation  executes  the  lesson 
plan.  Then  Instructional  Management  maintains  the  lesson  through  adding  and/or 
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deleting  topics  to  keep  it  current.  Finally,  Instructional  Evaluation  is  concerned  with  how 
effective  the  design  was.  Its  main  objective  is  to  find  weak  areas  to  repair.  (Reigeluth, 
1983) 

Of  Instructional  Design,  Development,  Implementation,  Management,  and 
Evaluation,  ID  has  the  most  impact  on  the  instruction  process  because  it  is  the  foundation 
for  the  development  of  instruction  (see  Figure  4). 


Figure  4:  Relationship  Between  Design,  Development,  Implementation, 

Management,  and  Evaluation  From  Reigeluth 


The  other  areas  are  dependent  on  design  as  design  is  dependent  on  them  for  input. 
But  the  other  areas  are  not  dependent  on  one  another.  They  may  build  upon  one  another, 
but  they  do  not  provide  input  into  one  another  like  design  does.  Design  inputs  into 
Development  by  providing  the  initial  plan  to  construct  something  such  as  web-based 
instruction.  Second,  Implementation  relates  to  design  by  incorporating  various  methods 
of  design  to  teach  a  course.  Management  manages  various  methods  of  design,  and 
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Evaluation  identifies  and  remedies  weaknesses  in  the  instructional  system.  (Reigeluth, 
1983) 

To  be  effective,  design  is  dependent  upon  the  other  processes.  Development 
provides  economical  information  to  the  design  process.  Some  designs  are  more  expensive 
than  others  to  develop  and  it  is  up  to  the  administrators  to  decide  what  type  of  design  to 
use.  Implementation  not  only  provides  an  economical  decision  (because  some  designs  are 
costlier  than  the  others  to  implement),  but  it  is  also  used  for  constraints.  For  example,  an 
innovative  design  may  be  economically  sound,  but  may  be  poorly  implemented. 
Management  also  provides  economical  input  into  the  design  process  because  if  the  design 
cannot  be  managed  properly  or  it  is  costly  to  manage,  the  design  is  then  rendered 
expensive  and  may  have  an  economic  impact  over  the  long  haul.  Evaluation  validates 
how  well  the  instruction  was  designed,  although  it  does  not  provide  direct  input  into  the 
models  and  theories  of  ID.  (Reigeluth,  1983) 

2.  Models  And  Theories  Of  Instructional  Design 

To  make  instructional  design  more  effective,  instructors  need  to  understand  what 
complete  set  of  strategies  provides  the  best- desired  outcome  under  given  conditions  or 
constraints.  Reigeluth  (1983)  describes  an  instructional  model  as: 

An  integral  set  of  strategy  components,  such  as:  the  particular  way  the 
content  ideas  are  sequenced,  the  use  of  overviews  and  summaries,  the  use 
of  examples,  the  use  of  practice,  and  the  use  of  different  strategies  for 
motivating  the  students. 


Eike  an  architect’s  blueprint  showing  many  different  designs  and  features  of  one 
type  of  building,  instructional  design  should  also  show  different  views  of  instructions  in 
order  to  achieve  the  desired  outcome.  Each  view  is  a  blueprint  of  what  instruction  should 
be  like. 

These  models  may  be  fixed  where  the  same  method  variable  is  prescribed 
regardless  of  the  students’  actions.  Or  they  may  be  adaptive  where  different  method 
variables  are  prescribed  according  to  the  students’  actions.  They  then  build  upon  a  set  of 
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principles  that  are  referred  to  as  instructional- design  theory  (or  instructional  theory). 
Instructional  theory  is  a  set  of  statements  that  take  the  form  of:  conditions- models- 
outcomes.  (Reigeluth,  1983) 

3.  Evaluating  Instructional  Design  Theory 

When  analyzing  instructional  design,  two  questions  should  be  asked.  Is  it 
instructional  or  learning  and  how  good  is  it?  First,  instructional  design  theory  focuses  on 
the  methods  of  instruction  and  how  to  manipulate  the  instructional  environment  rather 
than  on  learning  processes.  Furthermore,  is  it  a  theory  rather  than  a  model  or  a  list?  Does 
the  theory  focus  on  the  pattern  of  models- conditions -outcomes?  For  instance,  does  the 
theory  have  one  or  more  models  associated  with  it;  does  it  also  have  a  set  of  conditions 
under  each  model  to  be  used;  and  does  it  have  outcomes  for  each  model  under  each  set  of 
conditions.  (Reigeluth,  1983) 

After  determining  the  theory,  evaluating  it  for  “how  good  it  is”  is  the  next  step 
taken.  Some  major  criteria  to  measure  instructional  theory  are: 


Comprehensiveness .  Does  the  theory  include  strategy  components  from  all 
major  classes  of  methods?  Are  the  boundaries  and  limitations  clearly 
explained? 

Optimality  or  usefulness.  Are  there  better  theories  that  can  achieve  desired 
outcomes?  If  not,  analyze  the  weak  areas  on  how  they  can  be  improved.  If 
so,  can  they  be  incorporated  into  an  existing  theory  to  produce  a  better 
model? 

Breadth  of  application.  Consider  all  models  collectively  rather  than  as  an 
individual  model. 

Parsimony.  Is  its  degree  of  complexity  warranted?  Is  it  cost  effective? 
(Reigeluth,  1983) 
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4. 


Instructional  Media 


a.  Why  is  the  Instructional  Media  important  for  Instructional 
Designers? 

Instructional  design  is  delivered  through  Instructional  Media  (IM).  Reiser 
and  Gagne  (1983)  define  Instructional  Media  as  “every  physical  means  by  which 
instruction  is  presented  ta  learners.  It  is  every  physical  means  of  instructional  delivery 
from  the  live  instructor  to  the  textbook  to  the  computer  and  so  on  that  would  be  classified 
as  an  instructional  medium”  (Reiser,  2001,  p.  55).  In  this  research,  the  instructional 
medium  is  the  computer  and  the  World  Wide  Web;  or  the  Web  for  short. 

Previously,  Phillips  (1998)  stated  that  the  computer  was  used  as  a  training 
tool  to  fill  the  void  of  distance  learning.  Designers  created  tools  such  as,  computer-based 
instruction  (CBI),  computer- assisted  instruction,  or  computer-based  training  (CBT), 
which  provided  interaction  between  learner  and  the  content  that  met  instructional  goals. 
However,  these  tools  were  rigid  due  to  their  lack  of  openness,  flexibility,  and  the  learning 
environment  was  only  between  student  and  content.  (Khan  &  Ealy,  2000) 

The  Web,  on  the  other  hand,  provides  an  open,  dynamic,  and  flexible 
learning  environment  with  implications  for  countless  applications  with  respect  to 
education  and  training.  The  Web  allows  instruction  between  learner  and  content, 
instruction  between  learner  and  students,  and  among  the  learners  (Khan  and  Ealy,  2000). 
Also,  E-Eearning  via  the  Web  is  potentially  cheaper  and  more  productive  and  can  be 
delivered  with  more  timeliness  than  either  classroom  learning  or  traditional  computer- 
enhanced  teaching  (Meyer,  2001). 

Due  to  the  Web’s  ability  to  allow  people  to  communicate  economically 
and  relatively  quickly  with  each  other  in  nearly  every  area  of  the  world,  government 
agencies,  corporations,  and  universities  are  taking  advantage  of  this  instructional  medium 
by  placing  training  packages  and  courses  online.  Corporations,  such  as  American 
Skandia,  use  e- learning  to  train  and  reskill  its  workforce.  Instead  of  spending  days  away 
from  their  desks  sitting  in  a  classroom,  employees  can  log  into  the  company’s  training 
site  and  attend  classes  during  breaks,  lunch,  and  evenings  at  home  at  their  own  pace 
(Meyer,  2001).  The  American  Iron  and  Steel  Institute  (AISI)  believes  that  e- learning  not 
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only  saves  money,  but  can  also  reach  entry-level  engineers  who  cannot  obtain  permission 
to  travel  to  live  seminars,  to  help  them  keep  current  and  improve  their  engineering 
accreditation.  Along  the  same  lines,  the  American  Institute  of  Architects  (AIA)  provides 
the  same  services  for  architects.  Arizona  State  University,  British  University,  and  the 
University  of  Salford  in  the  U.K.  offer  undergraduate  as  well  as  graduate  programs  for 
students  who  are  working  full  time,  raising  families,  and  cannot  attend  classes  until  10 
pm.  When  they  access  the  learning  sites,  the  students  connect  from  Malaysia,  Europe,  or 
Australia.  Overall,  the  Web  provides  the  flexibility  for  learners  to  achieve  a  quality 
education  at  their  own  time  and  pace.  (Rosenbaum,  2001) 

b.  How  Instructional  Designers  Use  the  Instructional  Media 

Instructional  designers  should  always  take  into  account  the  person  at  the 
other  end  of  the  wire  and  how  designers  will  use  the  technology  to  enhance  instruction 
(Khan  &  Ealy,  2001;  Pallof  &  Pratt,  1999).  Eeamers  may  live  in  various  parts  of  the 
country  or  world  and  their  technology  may  vary.  One  student  may  have  the  most  recent 
technology  while  the  other  is  still  using  an  old  source.  This  may  cause  a  problem  for 
students  due  to  the  capabilities  of  each  system.  The  student  with  the  year  2001 
technology  such  as  a  Pentium  4,  1.4  GHz  can  download  items  and  connect  faster, 
whereas  the  student  with  the  slower  and  older  technology  such  as  an  X86  model  will  take 
longer  to  perform  these  same  actions. 

In  addition  to  these  problems,  instructors  need  to  learn  how  to  use  the 
technology  in  order  to  deliver  the  subject  material.  Will  the  institution  provide  sufficient 
training  for  instructors  to  develop  and  deliver  an  online  course?  Will  faculty  also  have 
trained  support  if  a  problem  arises  with  the  technology? 

Another  consideration  regarding  users  (instructors  or  students)  is  the 
amount  of  experience  they  may  have  with  technology.  Some  users  may  be  very  adept  and 
are  able  to  catch  on  to  the  new  technology  while  others  may  have  difficulties  with  the 
technology.  Eor  designers,  they  have  to  take  these  two  parties  and  their  levels  of  expertise 
into  consideration  in  order  to  produce  a  successful  product. 
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When  using  courseware,  software  developed  to  deliver  instructional 
material,  the  Web,  CD-ROM  programs,  or  video  conferencing,  instructional  designers 
recommend  that  ease  of  use  and  visual  appeal  be  taken  into  consideration.  Ease  of  use  is 
when  the  equipment  is  simple  to  operate  for  instructors  and  participants.  It  should  be 
transparent  where  it  helps  the  users  accomplish  their  tasks.  The  hardware  should  provide 
enough  power  and  memory  to  connect  to  the  Internet  and  make  navigating  the  course  site 
easier.  The  best  method  to  evaluate  a  system  is  evaluating  how  users  comment  on  the 
system.  Usually,  the  lack  of  comments  represent  that  the  users  are  satisfied  with  the 
system  because  it  represents  that  transparency  has  been  achieved.  Visual  appeal  refers  to 
the  ability  to  create  a  site  that  captures  users’  interests.  Providing  space  for  biographies 
allows  participants  to  express  their  personalities  and  ideas.  This  assists  instructors  and 
students  to  familiarize  themselves  with  one  another  and  strengthen  the  lines  of 
communication  by  putting  a  face  to  the  name  and  discussion(s).  Virtual  reality  reduces 
the  creation  of  a  (potential)  distorted  image  of  participants.  Video  games  have  been  doing 
this  for  years.  The  game  Thief,  developed  by  Looking  Glass  Studio  Creations,  for 
example,  provides  a  brief  tutorial  on  how  to  use  the  joystick  or  keyboard  in  order  to 
maneuver  successfully  through  the  various  levels  of  play.  Players  think  that  they  are 
playing  the  game  when  in  reality  they  are  only  learning  how  to  use  the  game.  When  it  is 
time  to  play  the  game,  players  are  already  familiar  with  joystick  or  keyboard  movements, 
the  rooms,  the  characters,  and  the  various  levels  of  the  game.  (Pallof  &  Pratt,  1999) 


E.  USABILITY  IN  WEB -BASED  EDUCATION 

Research  in  web- design  and  instructional  design  involves  many  aspects  of 
usability.  Nieslen  (1998)  refers  to  slow  loading  pages,  obscure  site  design,  and  poor 
support  for  navigation  as  serious  design  flaws  that  hinder  users  from  completing  tasks 
(Evans,  2000).  Hayes  (1998)  also  states  that  courseware  products  are  chosen  based  on 
technical  specifications  and  features  rather  than  on  the  usability  attribute  of  ease  of 
learning.  This  attribute  should  be  for  both  users  -  the  faculty  who  put  the  course  online 
and  the  students  who  take  the  course  via  the  web.  She  stresses  that  focus  for  course 
delivery  systems  is  not  given  to  user  requirements  and  clear  operational  criteria  are  not 
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used.  Decisions  are  often  made  based  on  the  most  recent  and  available  technical 
information  like  bandwidth.  Overall,  Hayes  discussed  how  usability  principles  can  be 
applied  to  the  web  for  evaluation,  and  how  they  are  currently  evaluated.  Before  her 
analysis  of  three  groupware  products,  she  took  into  account  her  target  audience  (faculty), 
their  goals  and  tasks.  Her  analysis  concludes  that  user  feedback  needs  to  be  consistent 
and  simple  so  users  can  easily  create  links  to  their  files  in  a  timely  manner  and  reduce 
memory  overload  (Hayes,  2000). 

Parlangeli  et  al.  (1998)  discuss  how  multimedia  systems  in  distance  education 
affect  learning.  Their  interest  was  in  how  learners  learn  to  use  the  computer  while 
learning.  Even  though  multimedia  systems  are  said  to  be  the  next  major  tool  that  “helps 
improve  learning,”  their  results  suggested  otherwise.  From  three  experiments  dealing 
with  heuristics  and  end-user  evaluations,  and  a  learner  assessment  based  on  a  long 
distance  class,  the  authors  found  that  users  mainly  experienced  navigation  problems, 
memory  overload,  lack  of  information  about  the  result  of  their  own  actions,  and  difficulty 
in  performing  the  same  actions  across  different  environments.  They  concluded,  “a 
difficult  to  use  hypermedia  system  can  negatively  affect  learning  performance” 
(Parlangeli  et  al.  1998). 

Stayner  and  Procter  (1999)  deal  with  navigation,  link  time,  and  downloading 
while  using  the  Web.  Even  though  this  study  did  not  specifically  deal  with  usability  and 
educational  design,  it  warrants  consideration  because  of  the  research  pertaining  to 
navigation  issues  in  instructional  design.  These  factors  may  influence  users’  choice  of 
which  links  to  follow.  Sometimes,  it  is  hard  to  anticipate  arrival  times  and  quality  of 
documents  when  navigating  the  Web.  Download  times  are  unpredictable  and  long  and 
quality  is  usually  “bad”  if  users  cannot  make  it  to  their  destination.  Conversely,  users  will 
wait  for  a  long  link  if  they  know  that  the  material  is  of  good  quality.  Finally,  users  have  a 
difficult  time  planning  strategies  because  the  Web  is  always  unpredictable.  In  order  to 
validate  these  assumptions,  they  tested  two  usability  issues:  1.  Content;  How  do  users 
determine  the  quality  of  a  document?  2.  Temporal  behavior  of  linking;  How  long  do  you 
wait  for  the  link?  They  found  that  users  looked  for  clues  such  as,  country  name,  and 
common  names  from  web  sites  to  determine  whether  they  should  proceed  navigating  the 
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site,  link  to  other  sites  for  more  information,  and  then  they  either  downloaded  items  or 
left  the  site  entirely.  Furthermore,  some  users  surfed  the  Web  in  the  early  morning  or  late 
at  night  when  run  times  were  better  when  less  people  access  the  Web. 

Jones  and  others  (1999)  attempted  to  test  the  usability  of  software  used  for 
education  technology.  The  main  concern  is  that  the  educational  software  is  sacrificed  for 
the  sake  of  usability  and  HCI.  They  agree  that  usability  and  educational  technology  are 
dependent  upon  one  another  for  a  web-based  course  to  be  successful.  However,  they 
believe  that  the  software  evaluation  process  should  have  more  input  from  educators  in 
order  to  produce  a  better  product  for  learners.  Comparative  studies  have  yielded 
information  about  what  has  occurred  and  not  how  learning  has  occurred.  They  believe 
that  there  is  a  need  to  understand  “how  the  learning  occurred”  rather  than  “what  has  been 
learned”  in  order  to  produce  better  software  for  education.  By  studying  the  outcomes 
through  interaction  and  outcomes,  software  developers  and  educators  together  can 
determine  how  usability  contributes  to  educational  goals. 

In  other  research  regarding  instructional  design  principles,  Berge  (1998),  Najjar 
(1998),  and  Weston,  Gandell,  McAlpine,  and  Finkelstein  (1999)  mention  usability  issues 
such  as,  technical  support,  bandwidth,  design,  and  navigation  as  principles  for 
instructional  design.  These  principles  show  how  instructional  design  is  dependent  on  an 
instructional  medium  like  the  Web,  CD-ROM,  or  courseware,  for  its  success.  Without 
taking  usability  into  account,  educators  sacrifice  the  quality  of  web-based  education.  The 
goal  for  these  tools  is  to  assist  the  users  in  presenting  a  course  and  taking  a  course.  If 
these  two  goals  cannot  be  met,  then  not  only  have  designers  failed  by  not  taking  usability 
issues  into  consideration,  but  also  the  institutions  that  purchase  these  tools  fail  because 
they  have  placed  the  burden  on  the  instructors  to  deliver  a  quality  education  via  the  Web 
with  a  poor  product. 
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III.  INSTRUCTIONAL  DESIGN  AND  USABILITY 

PRINCIPLES 


From  a  cognitive  point  of  view,  the  most  challenge  [sic]  is  to  bring 
together  the  educational  content  that  students  can  access  in  both 
synchronous  and  asynchronous  modes  and  to  unify  their  respective 
pedagogical  approaches.  This  merger  is  possible  if  an  integrated 
educational  approach  is  taken.  In  such  an  approach,  the  expositive 
teaching  and  active  learning  activities  should  be  combined  and  balanced 
according  with  [sic]  their  respective  pedagogical  objectives,  their  didactic 
efficiencies,  and  technical  constraints  (Latchman,  Salzmann,  &  Bouzekri. 
1999,  p.  252). 


A.  DERIVATION  OF  PRINCIPLES 

Squires  and  Preece  (1996)  argue  that  thinking  of  learning  and  usability  as 
independent  issues  leads  to  superficial  evaluations  of  educational  reviews  and  that  many 
teachers  are  not  trained  to  consider  usability  (Jones  et  al.  1999).  Usability  and 
instructional  design  are  different  fields  of  study  that  are  dependent  upon  each  other  for 
the  success  of  web-based  education.  HCFusability  is  a  procedure  that  is  used  to  build 
effective  systems  while  instructional  design  pertains  to  what  the  instruction  should  be  like 
(Reigeluth,  1983).  Combining  the  two  fields  creates  a  good  interface/system  so  that 
content  can  be  easily  accessed,  read  with  ease,  and  examined  with  ease.  Likewise,  the 
content  needs  to  be  just  as  interesting  in  order  to  capture  and  hold  the  user’s  attention 
regardless  of  how  the  system  functions.  For  web-based  instruction,  Paralangeli  et  al. 
(1999)  state  that  the  application  may  offer  relevant  and  useful  information,  but  if  the 
interface  is  difficult  to  use,  the  educational  success  of  the  system  could  be  jeopardized. 
When  interacting  with  [hypermedia]  educational  applications,  the  interface  guides  the 
student  through  an  educational  path.  The  student  thus  has  to  deal  with  a  double  learning 
process:  on  the  one  hand,  s/he  has  to  learn  how  to  interact  with  the  system.  On  the  other 
hand,  s/he  has  to  acquire  new  and  likely  difficult  concepts.  These  two  aspects,  namely 
learning  how  to  interact  with  the  system  and  learning  the  content  it  provides  are  not 
independent.  If  either  of  these  balances  is  disrupted,  students  will  more  than  likely  have 
difficulties  in  learning  via  the  web.  Instructors  may  be  able  to  provide  good  instruction 
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via  the  web,  but  if  they  are  not  trained  to  consider  usability  when  designing  a  course, 
their  course  may  be  rendered  ineffective  because  of  a  poor  interface  and  usability.  For 
these  reasons,  usability  and  instructional  design  must  be  taken  into  consideration  when 
instructors  put  a  course  online.  (Berge,  1998;  Hayes,  2000;  Jones  et  al.  1999;  Paralangeli 
et  al.  1999). 

Web-based  instruction  also  depends  upon  the  computer  and  web  design  for  its 
success.  In  traditional  education,  instructors  devise  course  objectives,  print  them,  and 
distribute  them  to  the  students  while  explaining  them.  In  web-based  instruction, 
instructors  devise  course  objectives  and  place  them  online.  Instead  of  the  instructor 
presenting  and  explaining  the  course,  the  computer  now  “converses”  with  the  students. 
The  computer  acts  as  the  interface  between  instructors  and  students.  It  does  not  translate 
“what  is  meant  to  be  said,”  rather  the  computer  presents  “what  is  being  said”  (Eisinger  & 
Smith,  2000,  p.  6).  To  assist  instructors  in  this  communication  process  between  the 
instructor  and  student  via  the  computer,  instructional  design  principles  were  collected  and 
derived  from  the  literature. 

The  literature  review  indicated  that  usability  might  substantially  influence  the 
quality  and  effectiveness  of  web-based  instruction.  The  readings  mentioned  “technology 
minimalism”  (Berge,  1998;  Najjar,  1998)  and  “plan  the  lesson  plan  to  the  user’s  ability 
skills”  (Eisenger  &  Smith,  2000).  Hayes  (2000)  even  stated  that  despite  published  reports 
comparing  online  delivery  systems,  “none  of  these  reports  contain  data  about  usability 
testing  of  the  products  reviewed.  Even  so,  the  phrase  ‘ease  of  use’  is  often  sprinkled 
throughout  the  reports.”  These  patterns  pertaining  to  both  usability  and  instructional 
design  appeared  throughout  the  readings. 

Since  traditional  education  is  embedded  in  the  education  process,  the  best 
approach  to  web-based  instruction  was  to  compare  similarities  between  traditional  and 
web-based  education.  By  breaking  down  a  traditional  course  into  very  general  parts,  the 
following  five  areas  were  created: 

•  Eirst  day  of  class  (welcome  to  the  course). 

•  This  is  what  we  are  going  to  do  (administrative). 

•  Reading  material  and  assignment  (presenting  the  material). 
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Assess  what  has  been  learned  (exam). 

Final  course  evaluation  (assess  the  course). 


• 

B.  FOUR  REQUIREMENTS  FOR  AN  EFFECTIVE  ON-UINE  COURSE 

1.  The  System  Needs  to  be  Reliable 

First,  a  system  pertains  to  the  software,  the  computer,  the  server,  the  amount  of 
bandwidth,  the  IT  (information  technology)  support  to  maintain  the  system,  and  the 
training  to  assist  instructors  in  developing  an  on-line  course.  These  components  ensure 
the  success  of  placing  a  course  online.  Second,  a  reliable  system  is  outside  of  scope  of 
this  work  because  we  (as  educators)  cannot  tell  students  to  buy  the  latest  and  greatest 
system.  We  have  to  hope  that  students  do  have  a  system  that  functions  well.  Whatever 
type  of  courseware  the  school  purchases,  the  instructors  and  students  alike  are  at  its 
mercy.  This  is  where  institutions  should  consider  the  minimal  requirements  for  web- 
based  courses  and  the  type  of  technology  the  users  will  use  to  access  the  course.  A 
conservative  estimate  of  the  technology  should  be  taken  into  account.  Usually,  the 
estimates  should  be  the  lowest  acceptable  technology  on  which  the  courseware  is  able  to 
function. 

2.  Tbe  System  Needs  to  be  Usable  to  Instructors 

Instructors  comment  that  time  and  the  lack  of  training  are  the  most  common 
problems  in  putting  an  online  course  together.  A  checklist  that  assists  instructors  with 
putting  a  course  online  could  alleviate  the  time  and  burden  instructors  have  with  putting  a 
course  online.  A  standardized  checklist  should  be  created  to  give  instructors  the  essentials 
foundations  in  creating  online  courses.  The  difficult  part  is  standardizing  a  generic 
checklist  to  help  instructors,  so  that  instructors  should  not  have  to  be  concerned  about 
usability,  HCI,  and  instructional  design.  Rather,  they  should  be  concerned  with  how  to 
put  the  math,  computer  science,  or  history,  course  on-line.  A  checklist  designed  to  cover 
these  three  areas  would  alleviate  the  problem. 
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3.  The  System  Needs  to  be  Engaging 

A  good  book  will  captivate  a  reader’s  interest  to  finish  reading  the  book.  Well- 
designed  eourses  are  no  different  from  a  good  book.  In  traditional  education,  good 
courses  are  routinely  ruined  by  poor  presentation  and  organization.  Students  loose 
interest  fast  and  eannot  wait  until  the  semester  or  quarter  ends.  In  traditional  education, 
the  students  can  support  each  other  through  a  peer  network  to  help  them  get  through  this 
type  of  class.  But  in  web-based  education,  since  students  are  alone,  they  will  turn  off  the 
system,  and  eventually  drop  the  elass  due  to  the  frustration.  To  avoid  this,  the  instruetor 
needs  to  present  an  organized  and  interesting  leeture.  A  well-designed  eourse  should  also 
engage  the  learner  in  aetive  learning  proeesses  or  aetivities.  The  learner  should  want  to 
study  due  to  the  organization  of  the  eourse  and  ease  of  aeeess  to  materials  for  learning. 

4.  The  System  Needs  to  be  Usable  to  Students 

Of  the  five  eharaeteristies  that  Nielson  (1998)  stated  --  ease  of  learning,  efficiency 
of  use,  memorability,  error  frequency  and  severity,  and  subjective  satisfaction  —  ease  of 
learning  and  efficiency  of  use  relate  most  to  web-based  instruction.  From  the  onset  until 
the  end  of  the  course,  students  learn  and  use  the  system.  Any  system  the  students  aeeess 
should  be  easy  to  learn  and  use.  After  overeoming  the  initial  diffieulties  in  learning, 
students  should  not  have  more  problems  when  using  the  system.  Memorability,  error 
frequency  and  severity,  and  subjective  satisfaction  depend  upon  learning  and  using  the 
system,  whieh  eome  after  time  and  through  repetition  of  use.  Only  after  students  learn 
and  use  the  system,  then  they  can  determine  how  “satisfied”  they  were  with  the  system. 
Ease  of  learning  and  efficiency  of  use  are  important  beeause  they  are  needed  to  get  the 
learner  started  and  remain  eonstant  for  the  remainder  of  the  eourse. 

This  is  not  to  assume  that  the  student  will  necessarily  learn  from  a  usable  system, 
but  that  the  system  should  help  the  student  to  aeeomplish  intended  goals  of  the  eourse. 

These  two  eharaeteristies  also  pertain  to  instruetors  beeause  they  have  to  put  the 
course  online.  Sinee  the  emphasis  of  this  thesis  is  plaeed  on  how  the  student  ean  rate  a 
eourse  with  a  measuring  tool,  the  eharaeteristies  will  be  diseussed  in  this  portion  of  the 
thesis. 
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Ease  of  Learning 


Students  learn  an  application  while  learning  new  material.  It  is  important  to  take 
this  into  account  because  new  and  experienced  users  alike  begin  learning  a  system 
through  trial  and  error  (Hayes,  2000).  This  means  that  students  are  more  apt  to  begin 
“clicking”  features  on  the  interface  without  reading  the  instruction  manual  or  the  “help” 
manual,  which  are  normal  and  curious  exploratory  procedures  that  users  possess.  When 
users  become  confused  or  lost  when  completing  a  task,  only  then  do  they  consult  “help” 
(Nielsen,  2000).  The  courseware  should  provide  the  organizational  structure  for  the 
course.  The  courseware  needs  to  be  simple  and  flexible  in  order  for  the  students  to  learn 
the  application  and  for  the  instructor  to  design  the  course. 


Efficiency  of  Use 


A  system  may  be  easy  or  difficult  to  use  after  learning  how  to  use  it.  The  users 
may  have  learned  how  to  operate  the  system,  but  continued  use  of  should  not  be  difficult 
or  counter-productive  (Wyard  &  Churcher,  2000).  If  users  have  to  continually  struggle 
with  a  system  while  learning,  users  may  develop  a  negative  attitude  towards  distance 
learning  -  something  that  should  be  averted  if  ease  of  use  is  taken  into  consideration. 
Firewall  access,  bandwidth,  navigation,  and  time  to  wait  between  changing  pages  are  a 
few  of  the  technological  constraints  that  can  render  a  web-based  course  ineffective. 
Users,  regardless  of  experience,  will  not  tolerate  these  problems  that  are  associated  with 
the  technological  medium.  Students  should  be  able  to  use  the  material  with  ease  or  very 
little  difficulty  regardless  of  user  ability.  (Berge,  1998;  Hayes,  2000;  Najjar,  1999; 
Weston  et  al.  1999) 
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C.  INSTRUCTIONAL  DES IGN  AND  USABILITY  PRINCIPLES 

By  combining  the  information  from  the  literature,  the  similarities  between  web- 
based  education  and  traditional  instruction,  and  the  four  requirements  of  delivering  a 
course,  instructional  design  principles  and  usability  principles  were  generated.  Table  1 
shows  the  principles  pertaining  to  web-based  education. 
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Examples  of  Usability  Principles  Pertaining  to  Web-Based  Instruction 


Present  the  material  in  simple,  understandable  paragraphs  (simplification) 
Keep  the  content  as  organized  and  consistent  as  possible  (consistency) 
Guide  the  students  from  one  location  to  another  (navigation) 


Examples  of  Instructional  Design  Principles  Pertaining  to  Web-Based 

Instruction 


Get  the  learner  started 
Present  the  subject  material  to  the  learner 
Let  the  learner  assess  the  subject  material 
Let  the  learner  assess  the  overall  course 
Let  the  learner  assess  the  usability  of  the  courseware 


Table  1 :  Instructional  Design  and  Usability  Principles 
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D.  INSTRUCTIONAL  DESIGN  AND  USABILTY  PRINCIPLES  DISCUSSION 


I.  Usability  Principles  Pertaining  to  Web-Based  Instruction 


Present  the  material  in  simple,  understandable  paragraphs  (simplification) 


“Reducing  complexity  for  users  is  still  a  very  difficult  aspect  in  interaction 
design”  (Hix  &  Hartson,  1993,  p.  35).  After  all,  the  computer  is  “lecturing”  to  the 
student  and  the  student  is  reading  from  the  computer.  Eisinger  and  Smith  (2000)  state 
“the  original  thought  behind  online  education  was  to  take  what  was  had  and  put  it 
online”.  A  customization  effort  was  then  needed  in  order  to  give  life  to  the  material, 
which  many  people  were  not  prepared  to  do  (Eisinger  &  Smith,  2000).  The  content  needs 
to  be  presented,  so  that  the  student  does  not  become  inundated  with  information.  Taking 
into  account  that  nobody  can  sit  in  front  of  a  computer  for  long  periods  of  time  without 
interruption,  dividing  the  learning  experience  into  smaller,  more  manageable  periods  of 
time  not  only  makes  sense,  but  it  is  easier  for  a  learner  to  schedule  the  lesson  and  retain 
information  (Crawford,  1999;  Eisinger  &  Smith,  2000).  The  time  for  reading  text,  for 
example,  varies  among  the  researchers.  Some  suggest  15  minutes  and  then  an  interactive 
exercise  (Eatchman  et  al.  1999),  or  20  minutes  and  then  an  example  of  the  text  (Berge, 
1998);  but  it  should  not  be  more  than  one- and- a- half  hours  [combine  reading  and 
exercises]  (Eisinger  &  Smith,  2000).  In  another  finding  related  to  multimedia,  “reading 
too  much  from  the  CD-ROM  was  tiring,  so  students  printed  some  of  the  material  and  also 
requested  that  the  instruction  be  in  printed  form”  (Jones  et  al.  1999).  In  all,  the  material 
should  not  be  presented  like  online  books  of  the  past  because  they  are  a  “well-tried  relic 
from  the  early  days  of  the  Internet”  (Aussenhofer,  1999,  p.  92)  that  were  difficult  to  read. 
Educational  material  should  not  be. 
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Keep  the  content  as  organized  and  consistent  as  possible  (consistency) 


This  principle  reinforces  the  simple  and  understandable  paragraphs  principle  by 
organizing  the  content  and  ensuring  its  consistency  for  clarity.  If  the  content  is  not 
properly  organized  from  the  beginning  to  the  end  of  the  course,  this  distracts  students 
from  learning  the  content  because  now,  students  are  spending  time  on  deciphering  the 
information.  “One  of  the  more  important  rules  in  designing  instruction  is  to  have 
consistency  among  content,  instructional  objectives,  and  student  practice,  all  leading  to 
the  evaluation  that  matches  the  practice  on  which  students  have  been  receiving  feedback” 
(Berge,  1998,  p.  73).  Organized  and  consistent  content  also  improves  the  chances  of 
students  comprehending  and  learning  material.  One  method  of  ensuring  good 
organization  is  “asking  if  content  is  relevant  (value  of  content),  reliable  (content 
accuracy),  and  up  to  date  (recovery)”  (Weston  et  al.  1999,  p.39).  In  traditional  education, 
when  instructors  update  their  courses,  they  add,  delete,  and/or  revise  the  content  as 
needed.  In  web-based  instruction,  instructors  should  ensure  that  the  content  is  neat  and 
current  because  the  content  “communicates  and  teaches”  students.  After  all,  the  content  is 
the  only  link  that  students  have  to  the  instructor. 

Although  the  content  may  be  clear,  organized,  and  presented  in  understandable 
paragraphs,  how  is  it  being  delivered?  Attention  to  detail  needs  to  be  taken  in  great 
consideration  when  putting  a  class  online  because  the  instructor  cannot  correct 
himself/herself  immediately  after  making  a  mistake  (such  as,  grammar,  spellings,  etc.). 
The  creativity  of  organizing  the  content  can  be  a  challenge  because  something  so  simple 
as  a  misspelled  word  can  distract  the  student  from  learning  (see  Appendix  A,  section  E, 
project  3:  Early  Analysis).  In  addition,  strange  fonts,  lack  of  indentation,  and  spacing  also 
cause  problems  for  students.  “It  is  important  to  ask  if  a  legible  font  is  used 
(typeface/fonts),  whether  numbering  systems,  headings,  indentation,  and  spacing  promote 
consistent  presentation  (format  and  layout)”  (Weston  et  al.  1999  p.  39). 

Strategically  placed  hyperlinks  also  play  an  important  role  in  affecting  how  the 
content  is  organized.  Hyperlinks  should  be  placed  at  the  end  of  paragraphs  after  the 
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presentation  of  content.  This  ensures  that  students  read  the  information  first  and  then  use 
the  hyperlink  to  enhance  the  subject  material.  This  method  serves  another  purpose.  For 
students  who  want  to  continue  reading,  they  can  continue  the  lesson  without  being 
distracted  by  a  hyperlink  in  the  middle  of  a  paragraph  (See  Appendix  A,  section  E, 
project  five:  Usability  Analysis).  For  the  students  who  want  to  enhance  their  knowledge 
base  on  that  particular  item,  they  are  able  to  “explore  information  bases,  discover 
relationships  for  themselves,  or  transform  and  organize  information  in  ways  compatible 
with  their  own  needs”  (Astleitner  &  Leutner,  1999,  p.  8).  Whether  students  click  on  the 
hyperlink  or  not,  they  have  read  the  subject  matter  without  interruptions  and  they  have 
the  choice  to  view  or  not  view  the  content  (Lund,  1997). 

Finally,  the  number  of  levels  for  linking  to  other  sites  should  not  matter  if  the  sites 
have  been  organized  properly.  When  Evans  (2000),  for  example,  developed  guidelines 
for  research-based  web  design  guidelines,  she  found  in  one  study  that  people  were  able  to 
search  for  information  faster  and  more  accurately  on  bigger  systems  than  a  smaller 
system  due  to  the  site  being  designed  to  effectively  guide  users  in  finding  information. 
The  results  of  the  study  suggest  that,  “whether  or  not  a  website  has  been  organized  in  a 
way  meaningful  to  its  users  is  also  a  key  influence  on  its  usability”  (Evans,  2000,  p.  305). 

Overall,  the  main  focus  of  organization  and  consistency  is  presenting  succinct  and 
clear  information,  which  focuses  on  cognitive  and  motivational  design  of  hypermedia  or 
web-based  instruction  (Astleitner  &  Leutner,  1999).  After  all,  if  the  rontent  is  not 
organized,  then  the  design  of  the  system  and  the  course  is  rendered  useless  (Hayes, 
1999). 


Guide  the  students  from  one  location  to  another  (navigation) 


Web-based  instruction  provides  numerous  benefits  such  as,  quick  and  remote 
access  to  information,  convenience,  adaptability  to  change,  and  speed  of  communication 
(Piotrowski  &  Vodanovich,  2000).  However,  common  design  flaws  such  as,  slow  loading 
pages,  obscure  site  design,  and  poor  support  for  navigation  renders  the  Internet  unusable 
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due  to  users  experiencing  delays  and  running  into  dead  links,  which  causes  users  to 
terminate  a  task  prematurely  (Evans,  2000;  McKenzie,  Mims,  Bennet,  &  Waugh,  1999; 
Nielsen,  1993;  Paralangeli  et  al.  1999;  Piotrowski  &  Vodanovich,  2000;  Stayner  & 
Procter,  1999).  These  actions  show  that  navigation  is  vital  to  the  success  of  web-based 
education.  Instructors  should  take  this  into  account  when  they  would  like  to  have  students 
visit  other  web  sites.  Navigating  is  the  highway  to  information.  Weston  and  others  (1999) 
define  navigation  as,  “how  the  student  moves  through  the  instruction  and  how  the 
instruction  is  designed  to  facilitate  understanding  of  organization  and  structure  of 
content.”  To  prevent  and/or  minimize  potential  navigation  problems,  instructors  should 
guide  students  through  lessons  by  directing  them  where  to  go  and  what  to  expect.  In 
other  words,  students  open  a  door  to  a  room  (lesson  objectives),  walk  around  the  room 
(read  the  lesson),  and  leave  the  room  via  a  different  door  (perform  an  exercise  or  take  an 
exam),  and  repeat  the  steps  (proceed  to  another  lesson).  Informing  users  about  the  next 
site  and  time  to  wait  to  connect  or  download  material,  improves  the  Quality  of  Service 
(QOS)  that  students  expect  from  a  course.  Sometimes  connecting  to  another  site  takes 
time.  Users  should  be  able  to  tolerate  delays  in  connecting  because  they  have  been 
informed  about  the  time  lag  (Stayner  &  Procter,  1999).  Once  again,  students  should  have 
the  choice  to  continue  go  to  the  other  web  site  or  continue  reading  the  lesson  (Lund, 
1997). 


2.  Instructional  Design  Principles  Pertaining  to  Web-Based  Instruction 

Get  the  learner  started 


Students  should  initially  meet  the  instructor  and  become  familiar  with  the  course 
when  they  begin.  The  purpose  and  goals  of  the  course  must  be  clear  from  the  beginning 
(Ross,  1999).  They  should  be  concise  and  brief.  A  description  of  the  “big  picture”  of  the 
course  should  be  sufficient.  The  overall  goal  is  to  capture  students’  attention  while 
avoiding  information  overload  (Crawford,  1993).  Be  creative  in  explaining  the  course. 
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According  to  Najjar  (1999),  “this  type  of  instructional  design  should  use  a  personal  style 
(e.g.  personal  pronouns,  names  of  specific  people,  direct  quotations,  vignettes  of  famous 
people)  rather  than  a  formal  style  to  stimulate  learner  interest.” 

Letting  the  students  feel  welcomed  may  encourage  them  to  become  familiar  with 
the  web  site  by  exploring  the  course  contents  before  taking  the  course  (Ross,  1999).  By 
the  same  token,  allow  time  for  an  introductory  lesson  on  how  to  use  the  web  site  allows 
students  to  relax  and  relieve  the  pressure  of  learning  the  computer  and  course  at  the  same 
time  (Astleitner  &  Leutner,  2000;  Paralangeli  et  al.  1999;  Weston  et  al.  1999).  Using  the 
video  game  Thief  again  as  an  example,  by  giving  initial  tutorials  on  how  the  game 
functions  allows  users  to  feel  more  relaxed  while  playing  the  game  because  they  have 
already  been  exposed  to  how  the  game  works. 

This  area  should  also  be  the  starting  point  and  return  point  for  students.  Inform 
students  that  the  Home  page  will  be  the  gathering  point  for  the  class.  Any  new 
announcements  or  upcoming  events  will  be  posted  in  this  area  as  the  course  progresses 
(Pallof  &  Pratt,  1999). 

Before  continuing  with  the  course,  students  should  also  know  what  is  expected  of 
them  and  what  the  instructor  will  provide.  Usually  in  traditional  education,  the  instructor 
explains  administrative  procedures  such  as,  course  objectives,  syllabus  or  class  schedule, 
and  expectations  on  the  first  day  of  class  in  order  to  provide  guidance  and  understanding 
for  the  duration  of  the  course.  These  routine  administrative  tasks  provide  a  starting  line 
for  students.  In  web-based  instruction,  these  issues,  along  with  how  to  navigate  the  course 
site,  assist  the  student  in  becoming  familiar  and  accustomed  to  the  course  site  and 
technology.  Finally,  web-based  instruction  needs  to  begin  with  the  end  in  mind.  Course 
objectives  should  be  established  to  determine  what  the  students  will  learn  and  what  skills 
and  abilities  they  should  take  with  them  when  they  finish  the  course  (Crawford,  1993; 
Pallof  &  Pratt,  1999;  Ross,  1999;  Weston  et  al.  1999) 

For  an  effective  web-based  syllabus,  weekly  topics  for  discussion  and  the  due 
dates  for  graded  assignments  (quizzes,  papers,  tests,  etc)  should  be  placed  in  the  syllabus 
section  so  that  students  understand  how  the  class  will  meet  objectives  because  “the 
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syllabus  section  is  the  road  the  students  will  take  to  reach  their  destination”  (Pallof  & 
Pratt,  1999,  p.  88).  The  syllabus  should  not  only  be  brief  to  avoid  information  overload, 
but  it  should  also  be  topic -driven  to  give  students  more  flexibility  to  use  the  school’s 
courseware.  For  example,  if  the  syllabus  contains  a  weekly  topic  discussion  with 
readings,  students  have  the  freedom  to  read  the  material,  asynchronously  post  their 
comments  on  the  comment  board,  and  participate  synchronously  and  asynchronously  in 
discussions  before  moving  on  to  the  following  week.  If  a  test  is  scheduled  for  that 
particular  week,  then  students  will  also  have  the  flexibility  to  take  the  test  any  time  before 
the  scheduled  deadline  (Latchman  et  al.  1999). 

The  flexibility  that  web-based  education  provides  can  be  abused  if  there  are  no 
defined  expectations  (Pallof  &  Pratt,  1999).  Course  expectations  define  the  parameters  or 
constraints  that  apply  to  successfully  fulfilling  course  requirements.  They  also  define 
what  is  and  is  not  acceptable.  In  this  section,  student  evaluation  is  explained.  For 
instance,  how  much  weight  do  assignments,  tests,  and  group  and  individual  participation 
hold?  Expectations  for  participation  should  also  be  clearly  defined  to  give  students 
direction  and  let  them  know  that  synchronous  and  asynchronous  participation  is  just  as 
important  as  participating  in  traditional  group  participation.  (Crawford,  1993;  Pallof  & 
Pratt,  1999) 

Expectations  also  include  course  conduct.  Conduct  is  vital  because  students  are 
more  dependent  on  one  another  to  complete  assignments.  If  there  is  a  team,  for  instance, 
and  one  team  member  is  regularly  absent  from  all  team  discussions,  then  the  team  may 
suffer  due  to  the  lack  of  participation  of  one  individual.  The  rest  of  the  team  is  dependent 
on  this  team  member  who  they  only  know  via  a  computer.  Assuming  that  particular 
member  hinders  a  group  project,  the  group  in  general  should  not  suffer  due  to  one 
individual.  Another  example  of  an  expectation  is  how  many  times  a  course  can  be  missed 
before  the  student  is  dropped  from  the  course  or  a  grade  reduced  due  to  the  lack  of 
participation  in  discussion  boards  or  coordinated  chat  groups  for  example.  By  the  same 
token,  students  should  take  responsibility  to  notify  the  instructor  when  they  will  be 
unavailable.  (Pallof  &  Pratt,  1999) 
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Along  the  same  lines,  the  instructor  should  define  his/her  role  in  the  course.  Will 
the  instructor  organize  and  present  the  material,  or  actively  engage  the  students  by 
participating  in  group  chats  and  discussions,  or  will  the  instructor  be  a  cheerleader,  letting 
the  students  teach  themselves  and  only  help  when  it  is  truly  needed?  The  instructor’s  role 
should  be  clear  for  students  to  understand  the  class  organization  and  how  it  will  operate. 


Present  the  subject  material  to  the  learner 


If  the  materials  are  not  presented  properly,  students  lose  interest.  “It  is  difficult  to 
attend  a  boring  class  and  even  more  difficult  to  attend  a  boring  web  site  that  contains  a 
boring  class”  (Ross,  1999,  p.  3).  At  the  onset,  students  understand  they  are  about  to  take 
the  course  and  begin  learning.  At  this  juncture,  “instructors  shape  the  course  by  providing 
a  content-rich  resource,  which  encourages  students  to  explore  new  avenues  of  learning” 
(Latchman  et  al.  1999,  p.  248). 

How  instructors  organize  the  text  influences  how  well  students  learn.  Najjar 
(1998)  recommends  combining  graphics  with  the  information  to  reinforce  the  content  and 
to  focus  students’  attention  on  the  subject.  To  reinforce  the  metaphor,  “a  pictures  tells  a 
thousand  words”,  students  take  time  to  interpret  what  has  been  read  through  pictures.  Not 
only  are  text  and  graphics  easiest  to  use  for  on-line  environments,  but  also  when  they 
match,  students  then  have  a  better  chance  at  learning  the  material  (Berge,  1999;  Najjar, 

1998) . 

Another  consideration  for  placing  text  on-line  is  how  students  read  web  pages. 
Hayes  (1999)  reports  that  people  do  not  read  web  pages  as  they  would  print-based 
materials.  Users  prefer  to  scan  text  rather  than  read  it  word  for  word  on  the  Web  (Hayes, 

1999) .  As  a  result,  the  overall  principles  really  need  to  be  taken  into  account  when 
placing  the  text.  For  example,  after  giving  a  brief  review  with  graphics  on  the  human 
nervous  system,  place  a  web  link  at  the  end  of  the  paragraph  for  the  students  to  explore 
more.  Viewing  the  web  links  can  be  either  mandatory  or  voluntary  depending  on  how 
much  information  the  instructor  wants  to  present. 
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As  in  a  traditional  environment,  instructors  should  encourage  students  to  be 
interactive  within  the  Web  community.  Kearshey  (1997)  states  that,  “the  single  most 
successful  elements  of  on-line  education  is  interaction  among  participants”  (Berge,  1999; 
Weston  et  al.  1999).  Depending  on  the  instructor,  this  method  can  be  synchronous  (chat 
rooms)  or  asynchronous  (discussion  boards)  or  both.  Engaging  the  learner  through 
incorporating  activities  such  as,  practice  and  feedback,  opportunities  for  reflection,  and 
problem  solving,  greatly  enhances  the  desired  learning.  Interaction  provides  an  outlet  for 
all  students  to  voice  their  opinions  (Astleitner  &  Leutner,  2000).  It  is  also  good  for 
collaborative  learning,  since  many  students  are  more  comfortable  with  contributing  to  a 
discussion  on  their  own  terms,  with  ample  time  for  reflective  responses,  rather  than  to 
being  put  on  the  spot  in  the  live  traditional  classroom  (Latchman  et  al.  1999;  Ciavarelli, 
2001).  With  on-line  learning,  the  shyer  students  now  can  express  their  opinion  without 
being  stifled  by  the  more  expressive  students. 

Interaction  also  helps  students  to  develop  their  logic  and  thought  processes  by 
constructing  through  rehearsal,  argument,  and  persuasion  on  discussion  boards  or  in  chat 
rooms  (Berge,  1999).  Instructors  should  have  the  students  put  their  thoughts  on  the 
discussion  board  for  other  students  to  see  and  for  peers  to  comment  on.  Along  the  same 
lines,  Jones  and  others  (1999)  affirm  that  through  interaction,  writing  messages  help 
clarify  students’  thoughts  by  developing  arguments  and  building  upon  other  students’ 
contributions. 

Bosco  (1986)  and  Fletcher  (1989,  1990)  examined  75  learning  studies  and  found 
that  participants  learned  material  faster  and  had  better  attitudes  toward  learning  the 
material  when  they  learned  in  an  interactive  environment  (Najjar,  1998).  When 
presenting  materials,  instructors  should  stress  interaction  and  make  their  lessons  more 
topic  and  discussion  driven  rather  than  just  having  students  clicking  on  the  site,  reading 
the  notes,  and  then  clicking  off  the  site  (Berge,  1998). 

Instructors  should  take  into  account  the  time  factor  and  information  overload 
when  they  present  materials.  Students  cannot  participate  effectively  in  chat  rooms  and 
homework  assignments  due  to  time  constraints  and  the  amount  of  work  given.  The 

amount  of  reading,  discussions,  and  assignments  should  be  distributed  evenly  for  students 
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and  the  amount  the  student  should  give  to  the  instructor  should  not  overwhelm  the 
student.  Information  overload  inhibits  students  from  learning;  thus  information  should  be 
given  and  performed  in  understandable  paragraphs  and  within  a  reasonable  time  frame 
(Crawford,  1999;  Ciavarelli,  1999). 

Let  the  learner  assess  the  subject  material 


This  area  demonstrates  how  well  the  students  comprehended  the  subject  matter. 
The  instructor  should  divide  the  course  between  quizzes,  tests,  term  papers,  individual 
participation,  group  participation,  and/or  project  assignment(s).  Najjar  (1999)  says  that, 
improving  student  learning  performance  results  from  the  test  matching  the  information 
that  was  learned  and  the  given  test  matching  the  expected  test.  When  the  instructor 
decides  on  the  type  of  assessment,  it  must  match  what  was  taught  in  the  course.  Like 
traditional  education,  students  do  not  enjoy  being  told  that  the  exam  will  be  on  subject  x 
and  the  instructor  gives  an  exam  on  subject  y;  or  the  lesson  was  on  identification  of  body 
parts  and  the  test  is  about  the  functions  of  body  parts.  In  all,  give  the  students  what  they 
expect  because  they  cannot  ask  the  instructor  immediately  via  the  computer  as  they  can  in 
traditional  education.  (Weston  et  al.  1999) 


Let  the  learner  assess  the  overall  course 


Feedback  is  essential  for  improving  a  poorly  designed  course.  The  main  concern 
should  be  how  the  students  experienced  the  course,  the  instruction,  and  the  online 
experience.  Giving  students  a  means  to  respond  via  e-mail  or  a  questionnaire  builds  a 
stronger,  cohesive  on-line  community,  since  the  students  have  more  input  and  feel 
“equal”  with  the  instructor  (Astleitner  &  Leutner,  2000;  Crawford,  1999;  Jones  et  al. 
1999;  Pallof  &  Pratt,  1999).  In  a  traditional  setting,  students  see  the  teachers,  and  make 
personal  comments  about  the  instructor  when  evaluating.  They  make  it  personal.  In  web- 
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based  instruction,  the  evaluation  should  reflect  the  course  material  rather  than  the 
personality,  resulting  in  better  feedback  for  the  instructor  to  enhance  the  course  for  the 
next  group  of  students  (Astleitner  &  Leutner,  2000;  Pallof  &  Pratt,  1999;  Rockwell, 
Furgason,  &  Marx,  2000).  Furthermore,  on-line  evaluations  may  be  more  effective 
because  the  quiet  students  may  contribute  more  anonymously  without  intimidation. 
(Berge,  1999;  Pallof  &  Pratt,  1999) 


Let  the  learner  assess  the  usability  of  the  courseware 


The  instructor  should  have  the  students  provide  evaluations  at  the  end  of  the 
course  to  determine  the  ease  of  access  to  the  program,  delivery  of  the  material,  and 
technical  support.  The  technology  should  meet  the  students’  needs  because  the 
functionality  of  the  system  is  just  as  important  as  the  design.  (Hayes,  2000;  Jones  et  al. 
1999;  Lund,  1997;  Pallof  &  Pratt,  1999;  Ross,  2000;  Weston  et  al.  1999;  Wyard  & 
Churcher,  2000) 

The  faculty  should  evaluate  the  technology  as  well  to  determine  how  well  it 
worked  for  them.  They  should  also  evaluate  for  additional  training.  They  should  critique 
how  well  the  institution  provided  assistance  to  the  instructors.  The  instructors  cannot  put 
a  class  online  without  assistance.  If  the  institution  is  determined  to  have  successful  web- 
based  education  courses,  the  institution  should  ensure  that  instructors  receive  the  support 
and  training.  Some  forms  of  assistance  are  online  help  from  the  vendors  of  the 
courseware,  tutorials,  guidelines  from  trainers,  and  workshops  (Hayes,  2000;  McKenzie 
et  al.  1999;  Pallof  &  Pratt,  1999;  Piotrowski  &  Vodanovich,  2000;  Rockwell  et  al.  2000; 
Weston  et  al.  1999). 
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IV. 


PILOT  EVALUATION  OF  PRINCIPLES 


To  test  the  principles,  a  pilot  test  was  conducted.  The  goal  of  the  test  was  to  find 
and  correct  deficiencies  in  the  principles.  The  project  began  by  identifying  and  defining 
the  problem.  The  Naval  Postgraduate  School  (NPS)  has  been  actively  developing  new 
online  courses,  but  lacks  an  effective  tool  to  measure  its  long  distance  program.  The 
solution  was  to  create  an  online  course  based  on  the  principles,  and  test  its  effectiveness. 
Instructors  and  students  were  the  intended  audience.  Appendix  A  contains  the  project  in 
its  entirety.  This  section  only  provides  a  brief  overview  of  the  project. 

A.  PROJECT  SUMMARY 

After  conducting  the  needs  analysis,  user  analysis,  and  task  analysis,  requirements 
for  designing  and  building  the  course  were  gathered  for  the  design.  Using  paper,  crayons, 
and  pencil,  a  design  of  various  interfaces  with  subject  content  was  made,  and  then  shown 
to  various  subjects  for  their  critique.  The  course  was  a  tutorial  on  using  various  Internet 
middleware.  The  tutorial  had  a  page  for  each  welcome,  administrative,  course  objectives, 
syllabus,  and  assignment  sections.  The  learning  lesson  was  also  included. 

When  this  design  was  shown  to  potential  users,  they  commented  on  how  the 
navigation  buttons  should  have  been  placed  in  a  different  area  rather  than  on  commenting 
on  the  content  of  the  course.  As  a  result,  various  questbns  on  usability  of  the  interface 
were  asked  to  determine  how  it  affected  usability  and  learning.  After  making  corrections 
based  on  the  peer  review,  the  design  was  transferred  to  the  school’s  courseware  for  a  live 
demonstration. 

The  criteria  for  evaluating  the  design  was  based  on  how  well  the  students  could 
navigate  the  site  by  following  instructions.  Since  the  school  already  had  a  proven  and 
usable  system  that  could  deliver  the  course,  the  researchers  decided  to  test  the  content  of 
the  course  to  see  if  the  design  also  affected  how  students  learn. 
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B.  METHOD  OF  EVALUATION 

Four  subjects  took  a  web-based  course  tutorial  regarding  Internet  middleware. 
The  subjects  had  at  least  intermediate  computer  experience.  Four  students  belonged  to 
Computer  Science,  Information  Technology,  Systems  Management,  and  Operations 
Research  curricula.  The  subjects  had  a  strong  background  in  using  computers  and  one 
student  was  also  taking  an  online  course  delivered  via  a  different  courseware  tool  that 
was  not  used  in  the  research. 

Since  the  course  material  could  only  be  evaluated  through  subjective  means,  the 
research  placed  emphasis  on  collecting  data  on  following  course  instructions,  mainly 
through  navigation.  The  subjects  were  encouraged  to  talk  and  think  out  loud.  As  they 
took  the  course,  they  talked  and  thought  out  loud  so  the  researchers  could  subjectively 
analyze  how  the  subjects’  understood  the  course  content.  The  course  time  was  30 
minutes.  During  this  time,  the  researchers  wrote  down  their  observations,  the  subjects’ 
comments,  and  actions  as  they  navigated  the  course.  When  they  finished  the  course,  the 
researchers  spent  20  minutes  with  the  subjects  to  evaluate  the  course  via  a  checklist.  The 
checklist  acted  as  a  survey  or  post  questionnaire.  After  each  interview,  the  researchers 
and  each  subject  evaluated  and  reviewed  the  checklist  to  obtain  better  results.  The  results 
were  based  on  how  they  viewed  the  course  when  they  were  taking  the  course.  Another 
reason  why  the  researchers  walked  the  users  through  the  questionnaire  was  because  it 
stimulated  dialogue  on  what  the  users  thought  of  the  course.  This  was  better  than  just 
“checking  the  boxes”.  Furthermore,  It  assisted  them  in  commenting  on  their  experience. 
All  in  all,  they  also  felt  more  relaxed  during  the  dialogue,  which  provided  the  researchers 
with  valuable  feedback.  After  each  session,  the  results  from  each  researcher  were 
discussed  and  compared.  One  researcher  usually  recorded  a  comment  or  behavior  that  the 
other  researcher  did  not  notice  during  the  interviews. 


C.  LESSONS  LEARNED  FROM  THE  INITIAL  EVALUATION 

I.  Instructional  Design  and  Usability  are  Dependent  Upon  One  Another 

As  each  subject  took  the  course,  they  commented  not  only  on  the  contextual 
errors  such  as,  misspellings,  grammatical  errors,  lack  of  font  and  indentation,  but  also  on 
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the  brightness  of  the  screen,  the  button  colors  were  to  bright  or  dark,  and  the  navigation 
was  difficult,  especially  when  the  subjects  left  the  tutorial  and  visited  the  web  sites.  The 
students  did  not  know  when  to  stop  navigating  and  return  to  the  tutorial.  Even  though  the 
emphasis  of  the  evaluation  was  placed  on  the  presentation  method,  usability  issues  crept 
into  the  project,  which  reinforced  the  belief  that  usability  and  instructional  design  are 
dependent  upon  one  another. 

2.  Flaws  in  the  Questionnaire’s  First  Draft 

The  students  had  difficulties  in  understanding  the  questionnaire  due  to  the  poor 
wording  of  questions.  They  did  not  accurately  reflect  what  was  intended  for  the  research, 
since  the  subjects  were  repeatedly  told  to  perform  various  actions  again  and  the  subjects 
also  commented  on  how  to  interpret  the  questions  for  answering. 

Another  mistake  the  questionnaire  contained  was  the  rating  system.  The 
researchers  did  not  have  experience  and  skills  required  to  develop  a  rating  system  To 
compensate  for  the  lack  of  experience,  the  researchers  used  phrases  such  as;  ‘1  =  strongly 
disagree’,  ‘2  =  disagree’,  ‘3  =  no  idea,  not  really’,  ‘4  =  agree’,  ‘5  =  strongly  agree,  I  feel 
great’ .  The  wording  confused  the  subjects  because  they  did  not  understand  ‘no  idea,  not 
really’.  Moreover,  no  previous  method  for  quantifying  ‘no  idea,  not  really’  existed.  As  a 
result,  the  initial  rating  system  distorted  the  accuracy  of  the  data  due  to  using  an  unproven 
rating  system. 

D.  QUESTIONNAIRE  REVISION 

I.  Sections  and  Wording  Revision 

The  instructional  design  sections  were  shortened  from  five  sections  to  four  parts. 
The  Administrative  and  Welcome  section  were  combined  into  the  Getting  the  Learner 
Started  section  in  order  to  provide  greater  clarity  for  the  users.  Also  the  wording  within 
the  questionnaire  went  through  numerous  drafts.  After  each  revision,  an  informal  review 
consisting  of  NFS  students  who  had  taken  web-based  instruction  and  NFS  instructors 
who  have  given  online  courses,  reviewed  the  questionnaire  for  more  revisions.  Due  to 


43 


their  input,  the  questionnaire  evolved  into  a  more- focused  and  accurate  tool.  Appendix  B 
contains  the  revised  and  completed  questionnaire. 

2.  Replacement  of  the  Rating  System 

To  make  the  instrument  accurately  reflect  the  measurement  dimensions,  the 
Questionnaire  Interaction  Satisfaction  (QUIS)  version  7.0,  developed  at  the  University  of 
Maryland  by  a  team  of  researchers  in  the  Human  Computer- Interaction  Interaction  Lab  to 
measure  users’  subjective  satisfaction  with  specific  aspects  of  the  human- computer 
interface,  was  adopted.  This  particular  questionnaire  addresses  reliability  and  validity 
problems  found  in  other  satisfaction  measures,  and  is  highly  reliable  across  different 
spectrums  of  interfaces  (Norman,  Schneiderman,  Harper,  &  Slaughter,  1998).  Appendix 
C  provides  a  deeper  insight  into  the  QUIS  survey. 

The  flexibility  of  QUIS  was  also  a  factor  in  using  the  questionnaire  because  there 
were  many  sections  that  the  project  did  not  use  and  they  were  easily  omitted  from  the 
Instructional  Design  -  Usability  Principles  questionnaire.  The  QUIS  dealt  specifically 
with  the  system  and  delivery,  whereas  the  Instructional  Design  -  Usability  Principles 
questionnaire  also  needed  to  specifically  focus  on  instructional  design  issues.  Adopting 
the  QUIS  format  provided  structure  for  the  Instructional  Design  -  Usability  Principles 
questionnaire  provided  this  flexibility. 

Another  appealing  aspect  of  the  QUIS  questionnaire  was  the  9-point  rating  scale. 
The  scale  goes  from  one  extreme  such  as,  ‘illogicar  to  ‘logical’.  The  midpoint  of  the 
rating  scale  (5)  can  be  used  as  a  criterion  representing  an  average.  Above  5  is  considered 
being  better  than  normal,  and  below  5  is  considered  less  than  normal.  By  plotting  the 
results  and  finding  the  mean,  the  reliability  of  each  variable  can  be  determined  as  well  as 
identifying  flaws  in  the  system  and  the  questionnaire.  If  there  are  unusual  rankings,  the 
graphing  will  assist  in  identifying  areas  for  analysis.  (Norman  et  al.  1998) 
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V.  METHOD  OF  EVALUATION 


A  trial  evaluation  was  conducted  to  evaluate  the  revised  Instructional  Design- 
Usability  questionnaire  as  an  effective  measuring  tool  for  web-based  instruction.  The 
subject  pool  consisted  of  resident  and  non-resident  students  from  three  classes  using  the 
Web  as  a  mode  of  instruction. 

A.  SUBJECTS 

The  first  class,  SS3011,  Space  Systems  Technology  and  Applications,  was  a 
purely  web-based  course  involving  students  from  NFS  and  throughout  the  United  States. 
The  second  course,  OC2022,  Scientific  Fortran  Programming,  consisting  of  five  students, 
was  a  web-enhanced  course,  where  students  went  to  classes  and  used  the  web  as  an 
enhancement.  The  third  class,  MN3384,  Acquisition  Production  and  Quality 
Management,  contained  two  sections.  The  first  section  consisted  of  resident  students  and 
the  second  section  was  delivered  via  Video  Teleconferencing  Environment  (VTE)  to 
students  in  Huntsville,  AE  and  Eort  Monmouth,  NJ  as  well  as  resident  students.  The  two 
sections  were  also  web-enhanced.  Of  the  48  students  from  both  MN3384  and  OC2022 
who  attended  lecture  and  used  the  Web  for  laboratories,  tests,  and  homework,  28  students 
took  the  survey. 

B.  INSTRUMENT 

The  Instructional  Design- Us  ability  questionnaire  consisted  of  83  questions 
involving  questions  based  on  the  background  material  and  derived  principles  involving 
instructional  design,  usability,  and  web  design.  Two  questions  related  to  demographics  in 
order  to  identify  the  classes  and  how  the  class  was  presented.  The  rest  of  the  questions 
pertained  to  the  questions  regarding  usability  and  instructional  design.  Appendix  D 
provides  the  questionnaire  used  in  the  survey. 
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C.  PROCEDURES 

1.  Administration 

The  questionnaire  was  administered  via  the  Web  in  June,  2001  to  the  students. 
The  personal  information  gathered  from  each  student  was  limited  to  the  name  of  the  class 
and  if  the  class  was  delivered  via  purely  web-based  or  hybrid.  There  was  no  other 
personal  information  collected  that  could  potentially  relate  the  participant  to  his/her 
responses. 

2.  Preparation  and  Evaluation 

Because  of  is  length,  the  time  to  take  the  questionnaire  was  estimated  to  be 
between  20-30  minutes.  A  brief  explanation  was  given  to  the  students  regarding  the 
importance  of  the  survey  and  asked  for  their  honest  opinion  about  how  the  course  was 
presented.  The  questionnaire  was  filled  out  on  a  voluntary  basis.  Twenty-eight  (28) 
students  responded. 

3.  Medium 

The  NPS  Office  of  Strategic  Planning,  Educational  Assessment  and  Institutional 
Research  (SPEAR)  converted  the  paper  format  of  the  questionnaire  into  a  web-based 
survey  through  the  software  Survey  Said.  The  software  creates  and  processes  surveys 
online.  Once  the  questionnaire  was  placed  online,  the  students  were  sent  the  URE  address 
to  take  the  survey.  The  answers  were  recorded  into  a  database  and  interpreted  using  an 
Excel  spreadsheet,  and  then  graphed  by  conditions  using  the  program  Stat  View. 
Responses  with  ‘  Not  Applicable’  were  omitted  from  the  calculations  to  provide  accurate 
data  for  the  average.  The  results  are  tabulated  in  appendix  D. 

4.  Evaluation  of  the  Questionnaire 

The  questionnaire  was  evaluated  in  part  by  how  the  students  rated  the  course.  The 
intent  was  to  demonstrate  the  derivations  of  the  average  means  from  each  class  above  and 
below  a  criterion,  and  then  make  an  analysis  of  how  the  courses  were  rated.  Since  the 
rating  scale  is  based  on  the  QUIS  7.0  system  of  1  to  9  (APPENDIX  C: 
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QUESTIONNAIRE  EOR  USER  INTERACTION  SATISEACTION),  the  midpoint  (5) 
was  the  criterion  for  a  neutral  rating.  If  the  rating  was  above  5,  then  the  rating  can  be 
considered  above  average  and  better.  If  the  rating  is  below  5,  then  the  rating  could  be 
considered  negative  if  the  questions  written  to  be  positive.  Eor  instance,  in  part  four  of 
the  questionnaire,  a  question  such  as  “there  were  difficulties  with  navigation?”  is  not 
negative  if  the  rating  is  a  2.  Regardless  of  rating,  the  questionnaire  attempted  to  reveal 
how  groups  rate  particular  parts  of  an  online  course. 
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VI. 


RESULTS  AND  DISCUSSION 


A.  RESULTS 

Appendix  D,  section  A  shows  the  questionnaire  used  for  the  survey  and  section  B 
contains  comparison  graphs  between  the  two  groups.  Graph  1  shows  the  average  mean  of 
each  section  of  the  questionnaire.  Graphs  2-5  show  the  cell  mean  of  each  question  from 
each  section  of  the  questionnaire. 

The  results  from  the  questionnaire  (Appendix  D,  graph  1)  indicate  that  the 
VTE/online  group  enjoyed  a  higher  degree  of  user  satisfaction  than  the  hybrid  group.  The 
standard  deviations  between  the  two  groups  were  between  1.5  for  the  first  three  sections, 
and  2.5  in  part  4.  In  sections  1  to  3,  both  groups  rated  the  course  positively.  However,  in 
part  4,  the  hybrid  group  rated  the  usability  of  the  courseware  more  pessimistically  than 
the  VTE/online  group. 

1.  Part  One :  Getting  the  Learner  Started 

Graph  2  shows  that  the  hybrid  group  had  more  difficulty  in  understanding  course 
concepts.  The  average  cell  mean  difference  from  questions  4-12  was  2.  The  class  syllabus 
and  course  expectations  were  not  as  clear  for  the  hybrid  group  as  they  were  for  the  VTE 
group.  Both  groups  agreed  in  question  13  that  the  content  from  the  web  site  was 
meaningfully  labeled.  However,  both  groups  differed  again  on  how  the  web  designs  of 
the  class  syllabus  and  course  expectations  were  presented  on  the  course  site. 

2.  Part  Two:  Presentation  of  the  Subject  Material 

Graph  3  shows  that  both  groups  were  consistent  in  their  ratings  of  questions  21- 
42.  Despite  some  exceptions  in  questions  29-30  and  questions  35-38,  where  the  VTE 
students  rated  portions  of  the  course  slightly  lower  than  the  hybrid  group,  both  groups 
answered  positively  on  the  instructional  quality  of  the  course.  However,  questions  44-51 
show  the  ratings  dropping  and  fluctuating  between  both  groups  where  the  VTE  students 
rated  lower  than  the  hybrid  group  and  then  returning  to  consistency  in  questions  51-53. 
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3.  Part  Three:  Assessing  the  Subject  Material 

In  graph  4,  both  groups  rated  consistently  high  with  the  exceptions  of  question  59, 
which  had  a  cell  mean  of  5  for  the  hybrid  group  and  6  for  the  VTE  group,  and  question 
61,  which  had  a  cell  mean  of  3  for  the  hybrid  group  and  1  for  the  VTE  group. 


4.  Part  Four:  Assessing  the  Usability  of  the  Courseware 

In  graph  5,  the  VTE  students  rated  higher  than  the  hybrid  group  in  questions  68- 
71.  However,  qiestions  72-80  show  the  VTE  group  cell  mean  average  dropped  to  2, 
whereas  the  cell  mean  average  for  the  hybrid  students  dropped  to  4.5. 


5.  Conclusions 

The  questionnaire  displayed  its  potential  to  be  used  as  a  checklist  for  instructors. 
In  areas  such  as  question  5  (appendix  D,  graph  2),  upon  review,  the  instructor  understood 
why  both  groups  rated  the  question  relatively  low.  This  is  an  example  of  how  the  rating 
scale  and  the  anchoring  words  provided  solid  measuring  support  to  reveal  positive  areas 
as  well  as  negative  areas. 

Despite  the  positive  areas,  the  questionnaire  also  revealed  a  limitation  by  not 
providing  evaluation  items  for  the  questionnaire  itself.  There  was  no  mechanism  added 
into  the  questionnaire  to  determine  if  the  questionnaire  is  an  effective  measuring  tool. 
Since  numbers  can  be  interpreted  many  ways,  there  needed  to  be  a  section  asking  the 
students  about  the  tool  itself. 

Eurthermore,  the  volume  of  low  responses  indicated  another  limitation.  This 
questionnaire  would  have  been  more  effective  if  there  were  more  participants.  One  class 
failed  to  take  the  questionnaire  due  to  an  administrative  error,  and  the  number  of  students 
from  another  class  was  too  low  to  use  as  a  case  study.  In  the  third  course,  MN3384,  24 
students  responded,  which  provided  a  good  assessment  of  the  course  and  how  it  could  be 
improved. 
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Since  the  students  came  from  either  resident  web- enhanced  (hybrid)  or  non¬ 
resident  (VTE  web-enhanced),  the  responses  provided  a  means  of  categorizing  and 
analyzing  the  results  based  on  these  two  conditions. 

B.  DISCUSSION 

The  data  collected  during  the  course  of  this  study  reinforced  the  instructor’s 
opinion  on  problems  that  he  encountered  with  his  web  site  during  the  course.  In  a  post¬ 
survey  interview,  trends  from  selected  questions  were  discussed  to  determine  what  were 
the  causes  and  solutions  to  the  responses. 

The  results  in  graph  1  (Appendix  D)  show  that  the  VTE  students  had  a  higher 
level  of  user  satisfaction  throughout  all  areas  of  the  questionnaire.  This  could  be 
attributed  to  the  instructor  not  being  physically  present,  thus  forcing  the  non-resident  to 
concentrate  more  on  the  material  than  the  resident  students.  Eurthermore,  the  VTE 
students  currently  work  in  the  field  of  acquisition  and  were  already  familiar  with  the 
lecture  material  from  the  course.  As  a  result,  they  could  easily  apply  lectures  and 
assignments  to  a  daily  routine  whereas  the  resident  students  were  in  the  process  of 
learning  new  material  and  lacked  the  work  experience. 

The  quotes  from  the  discussion  represent  the  thoughts  from  the  instructor  on  how 
he  viewed  the  ratings  of  the  two  groups. 

I.  Part  One:  Getting  the  Learner  Started 

The  VTE  students  felt  more  comfortable  in  getting  started  with  the  course  due  to 
initially  understanding  the  course  topic  better  than  the  resident  students.  There  was  a 
difference  of  opinion  between  the  groups  in  the  value  of  course  navigation  instructions 
and  biography  of  the  instructor.  The  VTE  students  felt  that  having  this  knowledge  was 
helpful  for  them.  This  extra  information  substituted  for  the  lack  of  the  instructor  being 
physically  present.  The  students  understood  the  instructor  and  pertinent  information  on 
navigation.  The  hybrid  students,  on  the  other  hand,  felt  that  this  information  was  not 
pertinent  to  them  due  to  the  instructor  being  physically  accessible  at  all  times.  Whenever 
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a  problem  arose,  the  resident  students  could  schedule  visits  and  talk  to  the  instructor 
before  and  after  class.  Through  informal  discussions,  the  hybrid  students  developed  a 
better  relationship  with  the  instructor. 

Despite  both  groups  agreeing  that  the  content  from  this  section  was  meaningfully 
labeled,  they  differed  again  when  rating  the  content  in  course  syllabus  and  course 
expectations.  “The  content  was  not  provided  in  small  paragraphs,  the  color  of  the  pages 
was  just  black  and  white  and  nothing  in  between.  They  were  not  labeled  well.  For 
example,  there  was  information  on  grading  and  assignments  but  there  was  not  an 
explanation  regarding  the  course  itself.  The  information  was  scattered  and  not  placed  into 
modules.  Now  there  are  objectives  and  explanations  placed  before  the  start  of  each 
module  and  reading  material.” 

Since  the  VTE  students  probably  had  taken  web-based  courses  previously,  it  is 
more  than  likely  that  they  have  already  seen  better  or  worse  course  sites.  They  also  take  a 
more  “we  will  get  through  this  philosophical  attitude.”  They  understood  and  accepted 
that  there  was  a  distance  gap  and  the  best  method  of  success  was  to  consult  with  one 
another  when  problems  arose  or  if  the  material  was  difficult.  For  the  resident  students, 
this  was  a  new  experience  for  them.  Learning  new  subject  matter  takes  time.  The 
vocabulary,  theories,  and  background  are  difficult  concepts  to  grasp  initially.  After  the 
first  test  or  assignment,  students  begin  to  feel  comfortable  because  they  have  established 
some  foundation.  The  resident  students  also  had  three  other  classes  to  worry  about 
whereas  the  VTE  students  only  had  this  one. 

2.  Part  Two:  Presentation  of  the  Subject  Material 

The  trend  from  each  group  was  consistent  with  what  the  instructor  did  not  do 
during  the  course.  Eor  example,  the  instructor  did  not  use  the  discussion  board  and  chat 
room  to  enhance  the  Web  portion  of  the  course.  The  instructor  incorporated  these 
changes  for  the  next  course.  He  will  use  the  discussion  board  as  a  tool  to  answer 
frequently  asked  questions  (EAQ)  and  make  available  a  student  self-assessment  from 
each  lesson  objective  where  the  students  are  able  to  discuss  objectives  to  provide  a 
“better  opportunity  to  grasp  the  material  for  both  sections.”  The  instructor  would  like  to 
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see  direct  and  succinct  comments  ranging  from  one -quarter  of  a  page  to  a  one -half  page 
relating  to  the  assignment.  He  believes  that  the  discussion  will  not  only  enhance  the 
students’  writing  skills  and  thought  processes,  but  also  “assist  the  bashful  students  in 
being  vocal.” 

In  addition  to  the  discussion  board,  the  instructor  has  adapted  the  chat  room  for 
students  to  share  their  ideas  through  interaction.  He  would  like  to  place  students  into 
small  groups,  consisting  of  a  mix  of  resident  and  long-distance  students  to  interact  and 
learn  from  each  other.  The  instructor  believes  that  this  mix  combines  the  strengths  of  the 
two  sections.  The  resident  students  are  the  users  and  experimenters  of  the  material  while 
the  non-resident  students  are  the  developers  who  possess  the  “real-world  knowledge” 
they  can  share  with  their  resident  peers.  “The  cooperation  between  the  two  groups  should 
equal  success.” 

Incorporating  these  two  methods  of  web-based  instruction  will  provide  the 
necessary  feedback  to  facilitate  a  successful  online  course.  The  instructor  feels  that  he 
should  receive  feedback  faster  to  adjust  to  the  concerns  and  needs  of  the  students.  “These 
are  new  methods  of  instruction  and  we  [instructors]  should  use  these  tools  to  help  us  pass 
on  the  knowledge  to  the  students  who  can  effectively  apply  it  when  they  return  to  their 
jobs.” 


3.  Part  Three:  Assessing  the  Subject  Material 

The  students  from  both  sections  did  not  have  a  mechanism  for  submitting  a 
critique  of  the  exams  to  provide  feedback  to  the  instructor.  Also,  there  was  a  lack  of 
feedback  given  back  to  the  students  to  review  incorrect  answers  from  the  exams.  In  the 
newly  revised  course,  the  instructor  has  an  area  for  students  to  comment  not  only  on 
exams,  but  also  on  written  and  reading  assignments.  The  area  is  anonymous  for  students 
to  “entice  students  to  speak  freely  without  retribution.”  In  all,  the  instructor  does  not  care 
who  gives  constructive  criticism,  but  that  there  is  a  way  to  receive  the  advice. 

Finally,  the  instructor  would  like  to  make  available  for  the  non-resident  students  a 
means  for  them  to  receive  feedback  on  the  final  exams.  The  resident  students  have  the 
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luxury  of  going  to  the  instructor’s  office  to  inquire  about  the  their  performance  and 
receive  additional  feedback  on  single  questions.  The  non-resident  students,  on  the  other 
hand,  have  no  means  of  physically  visiting  the  instructor.  This  situation  can  be  more 
difficult  and  frustrating  for  the  non-resident  students  if  they  desire  to  obtain  feedback. 
The  instructor  would  also  like  to  afford  an  area  for  the  students  to  have  access  to  review 
questions  as  they  study  for  the  exam. 

4.  Part  Four:  Assessing  the  Usability  of  the  Courseware 

The  VTE  students  experienced  fewer  problems  when  using  their  computers  to 
access  the  web  site.  This  is  significant  due  to  the  fact  that  during  the  course,  the  original 
web  site  was  not  accessible  to  due  to  firewall  problems  to  a  few  resident  and  non-resident 
students.  As  a  result,  the  instructor  hastily  constructed  a  basic  web  page  consisting  of 
assignment  folders  for  each  week.  The  resident  students  preferred  the  simple  web  page 
while  the  VTE  students  continued  to  access  the  original  web  site. 

Another  reason  why  the  VTE  students  had  better  user  satisfaction  was  because  of 
their  system.  The  VTE  students  used  the  computer  on  their  desks  while  the  resident 
students  experienced  problems  such  as,  “slow  downloads  and  navigations”  when  using 
the  computers  in  the  different  laboratories  across  campus.  The  VTE  students  understood 
their  computer  because  they  were  the  only  individuals  using  the  system.  On  the  other 
hand,  the  resident  students  could  not  ascertain  what  the  previous  student  did  to  the 
computer  before  the  resident  student  used  it.  This  has  been  a  problem  with  the  computer 
laboratories  on  campus  --  each  laboratory  is  different  and  each  computer  within  the 
laboratory  is  different.  This  arrangement  is  not  conducive  to  learning  because  the 
students  divert  their  attention  from  the  assignment  to  fixing  the  computer  problem. 

Having  more  experience  in  taking  web-based  courses  could  be  another  factor  in 
better  user  satisfaction  for  the  VTE  students.  By  interacting  more  with  the  computer  than 
the  resident  students,  the  VTE  students  felt  more  comfortable  in  handling  “technology 
problems”  associated  with  long-distance  learning. 
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VII. 


CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Based  on  the  results,  one  can  conclude  that  the  questionnaire  is  an  enabling  tool 
that  can  be  used  to  measure  the  effectiveness  of  instructional  design  and  usability  in  web- 
based  instruction  courses  even  though  the  questionnaire  was  not  fully  validated  by  a 
comprehensive  study.  The  trial  application  was  limited  to  test  an  initial  set  of  principles 
in  both  areas  of  study  and  to  establish  a  foundation  for  continued  research  into  measuring 
how  usability  and  instructional  design  are  not  independent  of  each  other  and  why  they 
must  be  taken  into  consideration  when  constructing  courses  for  the  Web.  The  data 
collected  from  the  hybrid  and  VTE  sections  and  the  post- questionnaire  interview  with  the 
instructor  are  consistent  with  the  background  information  and  the  initial  testing  of  the 
principles. 

The  design  for  the  questionnaire  was  based  on  input  and  feedback  from  actual 
users  who  had  experience  in  web-based  instruction,  human- computer  interface,  and 
usability  engineering.  Having  subject  experts  participating  in  the  design  assisted  in  the 
final  questionnaire  being  more  effective  than  the  original  one  constructed  for  the  initial 
testing.  Furthermore,  user  input  provided  more  validity  to  the  findings  and  helped  to 
support  the  principles  for  web-based  education.  The  rating  scale  and  anchoring  system 
from  the  QUIS  questionnaire  was  highly  instrumental  in  capturing  and  categorizing  the 
limited  data. 

Nevertheless,  the  questionnaire  contained  flaws  such  as  baiting  statements,  which 
assumed  that  there  should  have  already  been  something  in  place.  For  example,  a 
statement  such  as.  Instructional  examples  were  provided  insinuates  that  there  should  have 
been  examples  of  materials,  which  is  to  say  that  having  examples  of  materials  is  the  only 
correct  way  to  instruct.  This  is  false.  As  a  result,  this  caused  redundancy  because 
questions  were  repeated  but  with  different  phrasing  that  rendered  some  questions  to  be 
weak. 
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The  major  flaw  of  this  study  was  not  receiving  data  from  a  purely  web-based 
course.  The  survey  attempted  to  target  this  type  of  class.  The  resident  and  non-resident 
students  had  the  advantage  of  the  instructor  being  physically  present  vice  virtually 
present.  When  there  was  a  problem  such  as  the  firewall  connectivity,  it  was  remedied 
quickly  through  the  physical  interaction  with  the  instructor.  A  web-based  course  class 
does  not  have  this  advantage.  However,  the  error  of  not  obtaining  the  data  demonstrated 
that  communication  in  web-based  education  should  be  simple  and  direct  as  possible  to 
avoid  errors. 

Although  there  were  similarities  between  our  principles  and  principles  from  the 
readings,  the  major  difference  between  previous  studies  and  this  study  that  warrant  some 
attention  is  in  the  attempt  to  incorporate  human- computer  interaction  and  usability  into 
the  study.  These  issues  are  only  mentioned  in  the  readings  of  instructional  design,  but 
they  do  not  go  into  depth.  The  objective  was  to  see  how  web  design  and  the  usability  of  a 
system  could  affect  the  outcome  of  instructional  design:  Does  the  usability  of  a  system 
affect  the  educational  outcome  of  the  student?  In  this  study,  the  non-resident  students 
showed  a  higher  user  satisfaction  than  the  resident  students  in  computer  use  and 
comprehending  subject  material.  However,  the  percentages  of  the  grades  received 
between  the  two  groups  were  nearly  identical  (although  grades  were  not  used  as  a 
criteria).  Nonetheless,  the  data  shows  that  web-based  education  can  succeed  through 
good  instructional  design  methods  and  a  usable  system. 


B.  RECOMMENDATIONS 

Revising  and  retesting  of  the  questionnaire  is  necessary  in  order  to  improve  its 
effectiveness  as  a  measuring  tool  for  rating  web-based  education  courses.  One  area  that 
requires  improvement  is  devising  a  section  for  users  to  rate  how  user-friendly  the 
questionnaire  is.  Furthermore,  weeding  out  the  redundant  questions  such  as  questions  72 
and  74  will  streamline  the  questionnaire  by  not  only  simplifying  the  phrasing  of 
questions,  but  also  reduce  the  amount  of  ledundant  questions.  The  sequencing  of  the 
questions  should  be  kept  the  same. 
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Another  aspect  missing  from  the  questionnaire  were  questions  pertaining 
demographics.  Including  these  questions  would  assist  researchers  in  assessing  and 
identifying  different  types  of  groups  who  take  on-line  courses.  Identifying  the  different 
traits  should  provide  more  thorough  research. 

Retesting  the  questionnaire  on  purely  web-based  courses  will  ultimately  prove  the 
effectiveness  of  the  questionnaire.  Cooperation  between  NPS  and  deployed  units  would 
provide  a  strong  foundation  for  testing.  Deployed  ships  offering  training  and  classes  via 
the  Web  for  the  Marines  and  Sailors,  and  all  oversea  bases  offering  web-based  instruction 
courses  could  benefit  from  using  this  questionnaire  to  assess  their  on-line  courses,  while 
demonstrating  the  effectiveness  of  the  questionnaire  as  a  rating  mechanism. 


57 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


58 


APPENDIX  A.  INITIAL  EVALUATION  OE  PRINCIPLES 


To  test  the  principles  and  measure  the  effectiveness  of  the  Instructional  Design- 
Usability  questionnaire,  an  initial  evaluation  was  conducted.  The  project  went  through  a 
full  iteration  of  the  design- prototype- analyze  cycle  while  demonstrating  an  understanding 
of  usability  engineering,  evaluation  and  design  methods,  and  general  knowledge  of 
human- computer  interaction. 

Six  parts  were  performed:  1.  Problem  definition  2  Requirements  gathering  3. 
Design  4.  Prototype  5.  Usability  Analysis  and  lastly,  6.  Redesign.  Each  part  is  defined 
and  shows  the  work  performed. 

A.  PROJECT  ONE:  PROBLEM  DEFINITION 

This  phase  consisted  of  finding  and  stating  a  problem  in  one  sentence  for  clarity. 
An  explanation  was  given  for  what  activities  the  system  supported,  who  would  use  it,  the 
criteria  for  judging  the  system,  and  how  would  the  problem  be  approached. 

1.  Problem 

The  Naval  Post  Graduate  School  (NPS)  has  been  actively  developing  new  online 
courses,  but  lacks  an  effective  tool  to  measure  the  quality  instruction  in  its  long  distance 
learning  program. 

2.  Proposal 

a.  Solution  to  the  Problem 

Using  instructional  design,  human  computer  interface  (HCI),  and  usability 
principles,  a  web-based  prototype  course  will  be  designed  that  will  demonstrate  how  to 
create  an  effective  online  course. 

b.  Intended  Target  Audience 

This  courseware  is  web-based  and  can  be  accessed  from  any  location. 
Moreover,  the  intended  target  is  for  instructors  and  students.  The  instructors  should  be 
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able  to  put  a  course  online  with  very  little  difficulty,  and  students  should  learn  with  ease 
from  the  material. 


c.  Criteria  to  Evaluate  the  Project 

A  formal  evaluation  will  be  conducted  to  ensure  the  usability  of  the 
system.  The  criteria  for  judging  this  site  will  be  how  well  the  instructors  and  students 
navigate  the  site.  The  instructors  will  follow  a  set  of  instructional  design  principles  to 
build  the  course  and  the  students  will  navigate  the  course  via  instructions  and  take  an 
exam  to  demonstrate  their  acquired  knowledge.  The  students  will  then  evaluate  the 
course  with  a  questionnaire  that  contains  questions  regarding  how  the  content  was 
presented  and  the  usability  of  the  system.  The  main  consideration  for  the  evaluation  is 
ease  of  putting  a  course  online  and  the  ease  of  learning  online. 

The  procedures  to  evaluate  the  sessions  will  be  as  follows: 

Criteria  for  Instructors 

•  Time  to  complete  the  development  of  a  course  online. 

•  Number  of  faculty  failed  to  develop  a  course  online. 

•  Number  of  times  the  help  section  is  used. 

•  Number  of  times  user  becomes  frustrated  and  dissatisfied  with  the 
course. 

Criteria  for  Students 

•  Time  to  complete  a  course  online. 

•  Number  of  students  who  complete  the  course  exam. 

•  Number  of  times  the  help  section  is  used. 

•  Number  of  times  user  becomes  frustrated  and  dissatisfied  with  the 
course. 
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Based  on  these  criteria  and  the  user  questionnaire  regarding  instructional 
design  and  usability,  the  data  will  be  analyzed  to  correct  instructional  design  and  web 
design  problems. 

d.  Project  Design  Approach 

The  design  approach  throughout  the  project  will  be  a  customer- oriented  for 
students  as  well  as  faculty.  Since  the  courseware  is  already  complete,  a  prototype  will  be 
built  using  various  tutorials  regarding  Internet  middleware  from  students  at  NFS.  After 
each  design  phase,  informal  user  input  will  be  immediately  solicited  about  the  design. 
Students  from  various  curricula  will  evaluate  the  design  by  taking  the  course. 

The  conceptual  design  will  be  created  first  followed  by  the  semantic  design.  The 
physical  design  will  be  the  third  in  the  sequence  of  events  in  developing  the  course. 
During  this  phase,  a  rapid  prototyping  tool  will  be  used  to  develop  a  prototype  that  gives 
the  user  the  look-and-feel  of  the  actual  product.  Using  crayons  and  pencils,  the  design 
will  be  sketched  out  for  peers  to  evaluate.  After  redesigning  and  revising  the  design  of  the 
course,  the  course  will  be  placed  online  for  users  to  take. 

B.  PROJECT  TWO:  REQUIREMENTS  GATHERING 

A  needs  analysis,  user  analysis,  and  task  analysis  were  devised  to  understand  the 
requirements  for  the  user  and  the  system.  A  needs  analys  is  establishes  the  fact  that  a  new 
system  is  needed  due  to  external  demands  by  determining  basic  goals,  purposes,  and 
features  for  the  application.  The  goal,  situation  of  concern,  what  activities  do  the 
designers  support,  and  listing  and  explaining  the  features  of  the  system  are  issues  that 
assist  in  defining  what  a  user  will  do  with  the  system. 

A  user  analysis  produces  a  description  of  the  user  or  user  group  based  on 
significant  characteristics  and  limitations  that  might  affect  the  design  of  the  interface. 

Information  about  the  work  environment,  job  functions,  users’  tasks,  and 
characteristics  and  skills  are  taken  into  consideration.  Determining  how  the  system  will 
be  used,  how  often,  and  general  computer  skill  level  are  examples  of  defining  the  user 
analysis. 
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A  task  analysis  identifies  how  users  will  use  the  system.  A  detailed  description  of 
tasks,  subtasks,  and  methods  that  the  user  and  system  will  share  in  order  to  complete  a  set 
of  tasks  are  identified  through  a  top-down  decomposition.  The  purpose  of  this  analysis  is 
to  avoid  duplicating  a  set  of  current  procedures  that  the  user  and  computer  already 
perform. 

1.  Needs  Analysis 

a.  Goal 

Develop  an  interactive  interface  that  allows  instructors  to  develop  an 
online  course  and  students  to  learn  from  the  course. 

b.  System  Assumptions 

•  Site  is  accessible  from  any  location  in  the  world. 

•  Software  runs  on  a  different  server  from  the  NFS  main  server. 

•  Essay  questions  will  not  be  graded  automatically  due  to  various 
subjective  answers. 

c.  Instructor  Assumptions 

1.  Interface  automatically  displays  options  for  construction  of  course. 

•  Buttons  for  displaying  where  to  place  course  material. 

•  Instructors  begin  “clicking”  buttons  to  familiarize  themselves  and  feel 
comfortable  with  the  product. 

•  Instructors  begin  placing  material  on  the  site. 

2.  Instructor  does  not  have  any  knowledge  of  developing  a  course  online. 

d.  Student  Assumptions 

1.  The  interface  automatically  displays  options  for  navigating  the  site. 

•  Buttons  for  navigating  the  site. 
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•  Students  begin  “clicking”  buttons  to  familiarize  themselves  and  feel 
comfortable  with  the  product. 

•  Students  begin  learning  material  from  the  site. 

e.  System  Features 

1.  Provide  an  easy  interface  for  placing  material  on  online  for  an 
educational  course. 

2.  Provide  an  easy  interface  for  learning  material  from  an  online  course. 

3.  Allow  chat  rooms  for  discussion. 

4.  Allow  email  to  contact  students  and  instructors. 

5.  Allow  email  link  to  view  additional  sites  and  support  course  material. 

6.  Allow  welcome  board  for  placing  course  materials. 

7.  Allow  instructor  to  place  buttons  that  students  will  need  to  access 
information  for  the  course. 

8.  Allow  instructor/student  to  return  to  home  page  when  lost/confused. 

9.  Allow  instructor  to  view  course  material  after  placing  material  in 
respective  places. 

10.  Allow  instructor  to  modify  material  for  future  courses/exams. 

11.  Allow  instructor  to  form  various  exams  such  as,  multiple  choice, 
true/false,  essay,  fill-in-the-blank. 

12.  Allow  instructor  to  place  percentage  points  on  questions. 

13.  Allow  instructor  to  show  incorrect  and  correct  answers. 

14.  Allow  instructor  to  place  time  limits  on  exam. 

15.  Allow  instructor  to  place  time  limits  on  each  question. 
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2. 


User  Analysis  (Situation  of  Concern) 

a.  User  Group 

The  user  group  will  be  instructors  and  students.  These  users  possess  some 
elementary  computer  experience  such  as  web  browsing.  They  may  be  novice/casual  users 
who  will  not  tolerate  learning  how  to  navigate  a  site  while  attempting  to  learn  a  particular 
subject.  Therefore,  the  design  of  interface  must  be  so  basic  that  the  instructors  and 
students  can  perform  their  tasks  with  ease. 

b.  Online  Community 

This  application  is  designed  to  create  an  online  community  of  users  as  the 
users  learn  via  the  web.  The  users  must  feel  comfortable  using  this  site  while  they  are 
learning  online.  The  ease  of  this  courseware  allows  students  to  asynchronously 
communicate  with  each  other  through  chat  rooms,  email,  discussion  board,  and  the 
welcome  site.  These  features  should  strengthen  the  students  and  instructor  into  an 
intellectual  community,  so  they  can  take/support  future  courses  in  long  distance  learning 
via  the  web. 


c.  Simplicity  of  the  Interface 

The  users  can  be  expected  to  have  a  low  frustration  level.  The  interface 
should  be  kept  as  simple  as  possible.  All  buttons/options  should  be  easily  understood.  If 
the  users  spend  more  time  trying  to  navigate  the  system,  they  will  not  be  inclined  to  learn 
the  course  material  and  will  not  enjoy  learning  via  online. 

3.  Task  Analysis  (Activities  Supported) 

a.  Instructor  Activities 

Primary  Task  1:  Initiate  interface. 

•  Enter  user  name. 

•  Enter  user  password. 

•  Go  to  course  panel. 
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Primary  Task  2:  Understand  the  make  course  panel  interface. 

•  View  panel  options. 

•  Identify  which  buttons  perform  what  action. 

Primary  Task  3:  Develop  online  course. 

•  View  panel  options. 

•  Select  appropriate  option. 

•  Load  material. 

•  Select  view  material  to  review  material  from  “student’s  point  of  view.” 

•  Return  to  panel  options  and  repeat  steps  a-c. 

Primary  Task  4:  Allow  student  to  enter  course  site. 

•  Enter  student  user  name. 

•  Enter  student  password. 

•  Give  student  email  address,  username,  and  password. 

b.  Student  Activities 

Primary  Task  1:  Initiate  interface. 

•  Enter  user  name. 

•  Enter  user  password. 

•  Go  to  course  site. 

Primary  Task  2:  Understand  the  course  panel  interface. 

•  View  Welcome  site. 

•  View  panel  options. 

•  Identify  which  buttons  perform  what  action. 

Primary  Task  3:  Take  online  course. 

•  Select  appropriate  button  to  view  course  material. 

•  Read  course  material  and  perform  required  work. 
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Navigate  to  next  eourse  folder. 


Primary  Task  4:  Close  interfaee. 

•  Student  selects  file  menu. 

•  Student  selects  Exit. 

C.  PROJECT  THREE:  S\STEM  DESIGN 

This  portion  of  the  project  is  the  most  complex  due  to  unlimited  ideas  and 
possibilities  when  discussing  design  issues.  Carroll  and  Rosson  (1985)  explain  that 
design,  as  a  process  is  one  of  the  least  understood  development  activities- for  user 
interfaces  or  in  any  domain.  It  may  be  due  to  creativity  and  individuality.  Requirements 
become  lost  and  neglected  for  the  sake  of  creativity  and  individuality.  Due  to  this 
complexity,  constant  evaluation  of  design  is  necessary  in  order  to  develop  a  good 
product.  Conceptual  design,  detailed  design,  and  an  early  analysis  assist  in  maintaining 
design  as  a  process.  Conceptual  design  is  a  higher  level  of  design  and  pertains  to 
analyzing  objects,  attributes  of  objects,  and  the  relationships  between  objects,  actions  on 
objects,  actions  on  attributes,  and  actions  on  relationships. 

Detailed  design  or  visual  design  entails  activities  such  as,  wording  of  messages 
labels  and  menu  choices.  How  will  these  object  appear  on  the  screen  and  navigation  from 
screen  to  screen  are  considerations  that  designers  must  take  into  account.  A  simple  and 
effective  method  is  using  paper,  pencil,  and  crayons  to  sketch  the  screens  that  are  needed 
for  the  design.  Based  on  the  functionality  outlined  in  the  task  analysis  and  the  function 
points  in  the  conceptual  design,  designers  should  determine  how  this  functionality  will  be 
addressed  via  menus,  buttons,  dialogue  boxes  and  everything  else  that  will  be  used  for  the 
actual  version  of  the  finished  product. 

When  conducting  an  early  analysis,  designers  should  show  their  sketches  to  as 
many  reviewers  as  time  permits  who  are  unfamiliar  with  the  project.  The  designers 
should  walk  them  through  the  designs  and  tasks  to  obtain  some  initial  feedback  as  to  the 
efficacy  of  the  design.  Writing  down  the  comments  from  the  given  constructive  criticism 
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helps  designers  to  see  keep  their  design  according  to  the  specified  requirements  of  the 


user. 


1.  Conceptual  Design 

NOTE:  Designing  for  this  was  difficult  because  the  courseware  for  the  online 
course  was  already  created.  This  was  a  framework  that  the  researchers  could  not  change. 
Taking  this  constraint  into  account,  the  focus  of  this  phase  was  on  incorporating  the 
instructional  design  and  usability  principles  and  how  they  could  be  used  as  a  measuring 
tool  to  evaluate  how  effective  the  content  of  information  is  for  the  student. 

a.  Objects 

Student  Interface  Window  Buttons 

•  Announcements 

•  Course  and  Information 

•  Staff  Information 

•  Course  Documents 

•  Assignments 

•  Communication 

•  External  Links 

•  Student  Tools 

•  Resources 

•  Search 

•  Logout 


Instructor  Interface  Window  Buttons 


The  instructors  Interface  will  have  the  same  features  as  the  students  but 
they  will  also  have  a  control  panel  button  that  allows  instructors  to  build  a  course. 

b.  Object  Attributes 

The  students  and  professors  will  have  the  option  of  deciding  their  type  of 
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buttons,  colors,  and  shape  of  the  buttons. 


Student  attributes 


•  Username 

•  Password 

•  Registered  courses 

•  Age 

•  First  Name 

•  Last  name 


Faculty  Attributes 

•  Password 

•  Registered  courses 

•  Age 

•  First  Name 

•  Last  name 

•  Courses  to  be  taught 


Message  Attributes 

•  From  attribute 

•  To  attribute 

•  Subject  attribute 

Course  Attributes 

•  Course  number  with  departmental  code 

•  Course  title 

•  Description  of  course 

•  Goals  of  course 
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•  Prerequisites  for  course 

•  Professor  of  Course 

Assignment  Attributes 

•  Course  Name 

•  Due  Date 

•  Guidelines 

•  Assignment  Number 

c.  Relationship  between  Objects 

•  The  interface  window  contains  the  welcome  board 

•  User  begins  navigating  the  course  via  the  options  and  instructors 
comments. 

•  Only  instructors  have  access  to  the  control  panel  where  they  will  have 
numerous  options  to  build  the  course. 

•  Each  student  will  have  his/her  username  and  password  to  access  the 
site. 

•  Students  and  instructors  can  post  messages  for  assignments. 

•  Instructor  can  post  readings,  assignments,  and  exams  for  students. 

•  Students  can  submit  assignments  electronically  prior  to  due  date. 

•  Course  will  have  one  or  more  courses. 

d.  Action  Object  Attributes 

•  Users  are  able  to  navigate  the  site. 

•  Users  are  able  to  change  button  colors 

•  Users  are  able  to  change  shape  of  buttons  to  square  or  circle 

•  Users  are  able  to  navigate  the  site  and  return  to  Welcome  Page 

•  Users  are  able  to  enter  a  chat  room 

•  Express  ideas,  thoughts,  and  problems  about  assignments 

•  Eeave  the  chat  room 


69 


e.  Actions  on  Relationships 

•  Various  options  can  be  disabled  for  the  eourse  if  the  instruetor  deems 
neeessary. 

/.  Project  Concessions 

•  The  ehat  rooms  and  the  eourses  will  not  be  implemented. 

•  Material  from  one  eourse  as  a  design  will  be  shown. 

•  Coneentration  on  the  instruetional  design  prineiples  will  be  an 
emphasis  sinee  the  eourseware  has  already  been  ereated. 

2.  Visual  Design 

Note :  the  sketehes  for  this  portion  will  not  be  ineorporated  into  the  thesis  beeause 
the  design  eould  not  be  transferred  eleetronieally. 

After  entering  the  username  and  password,  the  announcement  page  displays  any 
news  the  student  needs  for  the  eourse.  Direetions  for  the  eourse  may  also  be  plaeed  in  the 
announeement  page.  The  eourse  material  appears  in  the  eourse  information  folder.  The 
student  ean  then  proeeed  to  navigate  the  eourse  by  ehoosing  various  options. 

A  tutorial  appears  after  the  student  elieks  on  the  eourse  information  folder 
allowing  the  student  to  read  and  study  the  material. 

The  large  magnitude  of  symbol  seleetion  and  ieon  plaeement  possibilities  makes 
it  diffieult  to  illustrate  these  via  sketehes.  Those  developed  by  the  low  fidelity  tests 
subjeets  are  ineluded  in  the  Early  Analysis. 


3.  Early  Analysis 

Note:  For  simplieity,  the  evaluator’s  Task  List  and  one  Low  Fidelity  Test  are  been 
submitted  as  examples. 

Four  test  subjeets  were  individually  asked  to  evaluate  the  eontent  of  material  and 
navigability  of  the  eourse.  Eaeh  subjeet  was  asked  to  read  and  then  explain  the  eontent  of 
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the  material.  From  these  explanations  and  more  suggestions  from  the  participants, 
improved  principles  and  a  measuring  tool  to  build  an  effective  course  were  derived. 


Summary  of  analysis 

•  The  content  was  presented  in  “small  understandable  paragraphs.” 

•  The  phrasing  and  wording  of  folders  such  as,  guidelines  and  organizing 
tasks  was  a  big  problem. 

•  Participants  commented  regularly  on  layout,  design  of  product,  and  its 
usability  rather  than  commenting  on  the  content. 

•  The  design  of  the  system  does  not  affect  how  experienced  or 
inexperierced  users  use  the  system  and  learn  from  the  material. 

•  From  evaluator’s  perspective:  The  instruction  design  principles  need  to  be 
improved  and  more  emphasis  on  creating  a  better  checklist  for  designing  a 
course  needs  to  be  implemented. 

These  results  will  be  considered  carefully  and  incorporated  into  an  improved 
design  before  proceeding  with  the  prototype  implementation. 
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Project  3  Task  List 

Evaluator’s  copy  of  the  task  list 

Reference  p.  50  of  Class  Notes,  Develop  the  Experiment 

Question:  How  do  we  dennnstrate  that  the  instructional  design  principles 
and  usability  principles  enhance  web- based  instruction? 

Questionnaire,  analysis  of  errors: 

1.  Student  understands  the  Welcome  Page  instructions.  Benchmark:  Student  does 
not  feel  “overwhelmed”  by  the  content.  (Information  is  presented  in  small 
chunks). 

a.  Read  instructions 

b.  Explain  instructions. 

2.  Student  is  comfortable  with  course  instructions.  Benchmark:  Student  navigates  to 
Course  Documents.  Student  does  not  feel  “overloaded”  by  the  content. 

a.  Read  course  objectives 

b.  Read  syllabus  (class  schedule) 

c.  Read  negotiating  guidelines 

d.  Explain  a  -  c. 

3.  Student  comprehends  presented  material.  Benchmark:  Student  reads  material, 
takes  a  quiz,  and  passes  quiz.  Student  does  not  feel  “overloaded”  by  the  content. 

a.  Read  material. 

b.  Explain  material 
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Project  3:  Example  of  Low  Fidelity  Test  Evaluation 

Tester:  Erich  Stefanyshyn 
Subject:  Male,  USMC,  Captain,  31  years 


Overview:  The  evaluation  lasted  25  minutes.  During  this  time,  the  subject  was 
given  various  outlays  of  the  interface  and  asked  to  review  the  content  of  material  on  the 
“screen.” 

Subject’s  first  impressions: 

Task  1:  Student  did  not  feel  “overwhelmed”  by  the  material.  He  thought  that  it 
was  organized  and  could  understand  the  intent  of  the  course.  However,  He  wanted  to  see 
the  message  once.  He  thought  that  a  “E'  time  for  visitors”  message  should  be  placed  to 
avoid  redundancy  and  boggle  of  material. 

Task  2:  Student  did  not  feel  “overloaded”  by  the  material.  However,  the  content 
could  be  better  explained.  For  example,  when  meeting,  the  instructor  needs  to  explain 
that  the  meeting  will  be  “meeting  online  asynchronously”  vice  “meeting 
asynchronously.”  Another  good  recommendation  is  meeting  synchronously  “via  chat 
room  and  discussion  boards  at  your  (groups  convenience).”  Furthermore,  there  should  be 
some  directions  on  where  to  navigate  next  from  the  course  objectives  folder  to  the  class 
schedule  folders.  It  helps  to  “channel  the  transition  from  folder  to  folder.” 

Task  3:  F‘  Part:  The  material  in  the  explanation  folder  was  not  overwhelming,  but 
it  was  not  organized  as  well  as  the  previous  Task  1  and  2.  For  example,  there  should  be 
check  boxes  assigned  to  the  instructions  to  remind  users  (inexperienced  and  experienced) 
where  to  go  and  what  to  do.  When  they  finish,  they  can  check  the  boxes.  The  objectives 
and  assignments  should  be  simpler.  Also,  place  time  limits  on  the  reading  assignments 
and  keep  a,  “instant  review  sheet  next  to  the  assignment  for  venting  purposes.”  For 
example,  if  an  assignment  lasts  one  hour  and  the  student  takes  2  hours  to  complete  the 
reading,  the  student  should  go  back  to  his  checklist  and  be  able  to  let  the  instructor  know 
that  there  the  assignment  was  “overload.”  Place  a  message  on  top  for  students  to  critique 
honestly  to  let  them  know  that  this  is  their  site  and  education. 
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2“*^  Part:  This  should  be  the  “last  hyperlink  beeause  it  takes  the  most  time  to  do.” 
Content  is  suffieient,  but  there  is  too  many  paragraphs.  Ineorporate  more  “bullets”  for 
both  users.  For  “inexperieneed,  to  see  ideas  and  experieneed  to  go  direetly  to  the 
material.”  Also  “hyperlink  the  bullets  for  the  paragraphs”.  Overall,  the  buttons  for 
navigating  are  eonfusing.  For  instanee,  Assignment  button  =  test  and  should  be  a  daily 
task  like  course  documents.  Also,  divide  the  lessons  into  day/time  periods.  For  example, 
Weekl:  day  1,  1  hour  limit,  day  2,  1  hour  limit,  etc. 

Subject’s  Opinion:  Subject  noted  the  need  to  change  some  of  the  buttons  for 
better  clarification.  Subject  stressed  that  the  material  needs  to  be  as  clear  and  concise  as 
possible  to  avoid  confusing  the  student.  Subject  would  also  like  to  see  the  live  program. 

D.  PROJECT  FOUR:  PROTOTYPE 

Based  on  the  results  from  the  design  in  Phase  3,  a  prototype  is  built  for  more 
testing  and  evaluation.  It  involves  production  of  at  least  one  early  version  of  the  system 
that  displays  the  important  features  of  the  system  in  operation.  Prototyping  is  very 
beneficial  to  a  project  because  it  involves  user  participation  and  lets  developers  observe 
how  the  users  interact  with  the  system.  In  short,  like  a  car,  the  user  is  “taking  the  system 
for  a  spin”  (Hix  &  Hartson,  1993). 

A  prototype  for  the  web-based  evaluation  was  constructed  using  the  school’s 
courseware.  A  course  based  on  Internet  Middleware  was  put  online.  The  lesson  involved 
Web/Site  Management  Tools  for  constructing  Web  pages.  It  was  built  according  to  the 
measuring  tool  that  was  made  following  in  Project  3  and  issued  in  project  5. 

E.  PROJECT  FIVE:  USABILITY  ANALYSIS 

A  usability  analysis  allows  users  and  developers  to  see  how  well  the  design  and 
the  prototype  were  constructed,  and  measure  its  effectiveness  and  usability. 

Planning  well  for  a  usability  analysis  is  imperative  for  its  success.  The  testers 
should  ensure  that  a  set  of  tasks  have  been  planned  for  the  users.  The  tasks  should 
measure  what  the  system  is  supposed  to  be  used  for.  Users  who  will  interact  with  the 


74 


system  should  be  chosen  as  subjects.  The  more  subjects  tested,  the  more  feedback  can  be 
garnished  for  analysis  to  assess  the  usability  of  the  system. 

At  this  stage,  data  collection  is  more  formal  and  contains  both  qualitative  and 
quantitative  elements.  The  same  questions  asked  in  the  phase  three  are  asked  more  in 
depth.  Evaluators  are  looking  for  where  the  system  did  not  meet  user  expectations. 
Efficiency  is  one  method  of  evaluation.  How  optimal  did  the  users  perform  the  tasks? 
How  long  did  it  take  to  complete  the  tasks?  Could  these  tasks  been  made  any  easier? 
Were  there  delays  in  the  system  (where  the  user  spent  time  thinking  about  what  to  do 
next)?  How  can  these  be  removed  or  minimized? 

Errors  are  also  analyzed.  Do  users  make  errors?  If  so,  can  they  be  engineered  out 
of  the  system?  If  not,  has  the  design  been  adequately  accounted  for  them  by  being 
recoverable  and  informative  messages  placed  to  assist  the  user?  What  errors  were  made 
and  were  they  critical? 

Learnability/Memorability  is  taken  into  account  by  evaluating  how  easy  the 
system  was  easy  to  use.  Will  users  need  help  and  training?  Most  importantly,  how  well 
does  the  error  rate  drop  as  the  user  become  more  familiar  with  the  system?  If  not,  the 
problems  need  to  be  identified  and  corrected. 

Finally,  user  satisfaction  with  the  system  is  also  evaluated.  Would  they  like  to  see 
it  on-line  for  real  usage?  What  changes  did  they  suggest  to  make?  Usually  the  user’s 
suggestions  bring  out  ideas  that  were  not  thought  about  previously.  Finally,  the 
measurement  of  user  satisfaction  is  when  the  user  does  comment  negatively  about  the 
system  and  does  not  say  anything  while  using  the  system.  This  demonstrates  that  the  user 
is  heavily  engaged  and  the  system  is  easy  to  use. 

1.  Results  from  the  Usability  Analysis 

The  Test  Tasks  outlined  in  Appendices  1  and  2  were  designed  to  observe  and 
record  how  the  users  interacted  with  the  presented  material.  Some  of  the  interactions 
involved  mouse  and  button  manipulation  based  on  instructions  within  the  content. 
Appendix  3  demonstrates  the  post  course  questionnaire  that  the  users  evaluated  after 
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taking  the  course.  When  users  did  not  know  what  to  mark,  they  were  referred  to  the  Task 
List  assist  them.  For  instance,  the  first  checklist,  “Are  the  students  welcomed  properly,” 
one  subject  did  not  know  what  to  put,  and  when  shown  where  the  welcome  phrase  was 
located,  he  was  ready  to  mark  5.  But  he  was  told  to  judge  the  guideline  when  he  was  at 
that  page  during  the  course  session.  He  changed  his  opinion  to  3  because  “he  did  see  it 
but  he  did  not  feel  as  welcomed  because  it  did  not  stand  out  and  grab  him.”  Finally,  at 
the  beginning  of  the  each  interview,  the  importance  of  comments  and  criticism  of  the 
interviewee  was  highly  expressed  and  encouraged.  Throughout  the  test  and  interview,  the 
evaluators  constantly  encouraged  the  users  to  comment  and  think  loudly. 

Also  contained,  as  appendices  are  the  consent,  minimal  risk,  privacy  act 
statement,  and  participant  demographics  forms. 


a.  Efficiency 

Were  the  users  able  to  navigate  based  on  the  instructor’s  comments? 

The  subjects  could  navigate  on  the  instructor’s  comments.  After  reading 
the  instructions,  the  students  spent  very  little  time  to  find  the  buttons  and  utilize  the 
system.  A  typical  question  during  the  process  was,  “Is  this  the  button  to  press?”  or  “Do  I 
go  here?”  After  clicking  on  the  button(s),  the  subjects  immediately  understood  how  the 
system  worked  and  could  concentrate  on  the  material. 

To  better  the  course,  all  subjects  preferred  that  a  hyperlink  be  placed  next 
to  the  final  instruction  instead  of  having  to  search  for  the  button.  Otherwise,  the  content 
was  straightforward. 


Did  the  computer  interface  interrupt  the  course  content? 

As  long  as  they  could  follow  the  course,  the  subjects  felt  that  the  system 
was  easy  to  use,  and  that  they  could  concentrate  on  the  material.  Since  none  of  the 
subjects  used  Help,  we  saw  that  as  a  good  indication  that  they  were  concentrating  on  the 
material  vice  fighting  the  system. 
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Did  the  users  feel  that  any  tasks  could  have  been  easier  to  perform? 


During  the  presentation  of  the  material,  as  the  subjects  read  and  clicked  on 
various  web  sites,  they  would  have  liked  to  have  seen  the  reading  objectives, 
assignments,  and  review  questions  on  the  reading  pages  in  addition  to  being  in  the 
Course  Syllabus. 

The  additional  links  within  the  material  should  be  placed  after  the 
paragraph  because  the  readers  preferred  to  read  the  material  first  and  then  have  the 
“option  of  clicking  the  link  for  more  information.”  One  subject  wondered  about  the  use 
of  links  when  she  said,  “this  could  go  on  forever,  when  do  I  stop?”  Placing  the  links  after 
the  paragraph  would  structure  the  material  better  and  let  the  students  read  the  main 
content  and  then  assists  students  who  are  interested  to  visit  other  sites.  The  uninterested 
students  could  continue  to  the  next  paragraph  of  information.  One  subject,  for  example, 
did  not  click  on  any  hyperlinks  because  he  did  not  know  where  he  would  go  and  also  did 
not  know  if  he  should  have  clicked  on  them. 

Overall,  the  learning  material  was  overwhelming  due  to  the  lack  of 
structure.  For  instance,  “The  web  sites  had  a  lot  of  information,  and  at  what  point  do  I 
stop  [navigating]?”  The  content  should  be  organized  better  and  an  explanation  the 
websites  should  be  provided  to  the  students.  This  way,  students  will  know  what  to  expect 
from  the  site  and  when  they  should  stop. 

b.  Errors 

Could  context  errors  be  engineered  out  of  the  system? 

All  subjects  complained  about  word  misspellings.  “It  distracts  me  from  the 
reading.  Now  I  have  to  start  all  over  again.  I  don’t  like  typos  in  my  textbooks  nor  do  I 
like  them  on  my  computer.” 

Bolding  was  another  issue  the  subjects  mentioned  when  they  navigated  the 
site.  The  wording  in  each  folder  was  identical.  Nothing  “stuck  out  and  grabbed  my 
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attention.”  At  the  Welcome  Page,  although  there  was  a  “welcome  comment,”  it  was  hard 
to  distinguish  because  it  was  in  plain  text  and  meshed  in  with  the  other  material.  As  a 
result,  two  of  the  subjects  “did  not  feel  welcomed.” 

The  biggest  complaint  involving  errors  was  the  lack  of  indentations  in  all 
folders  (especially  the  Course  Guidelines  folder).  We  failed  to  indent  the  material  and  it 
made  the  content  difficult  to  read  and  understand. 

Did  the  system  account  for  errors  ? 

No,  this  where  the  system  falls  short  of  expectations  and  the  instructor 
should  compensate  for  them.  The  system  itself  did  not  allow  for  indentation,  spelling 
check,  and  bolding.  When  we  tabbed,  the  cursor  went  to  another  location  on  the  screen 
Even  though  we  could  have  used  the  space  bar  to  compensate,  we  did  not  because  it  was 
time  consuming.  Nevertheless,  by  not  investing  the  extra  time,  and  effort  caused 
problems  for  our  users.  This  lack  of  system  functionality  causes  hardship  on  the 
instructor  (spending  time)  and  students  (dealing  with  the  problems). 


What  critical  incidents  could  be  identified? 

One  potential  (and  greatest)  error  associated  with  the  course  is  content 
instructions  and  linking  to  other  websites.  If  the  instructions  are  not  clear  and  the  web 
sites  are  not  explained,  students  could  spend  endless  hours  and  wonder  when  to  stop. 
They  could  become  frustrated  with  the  system  and  the  course,  and  eventually,  lose 
interest  with  the  online  education.  “The  web  site  should  be  used  to  explain  and  enhance 
the  material.”  This  avoids  information  overload  and  makes  learning  easy. 

c.  Learnability/Memorability 

Was  the  system  easy  to  learn  ? 

The  system  was  easy  to  learn.  The  users  did  not  have  complaints  about  the 
product.  The  wording  of  the  button  features  was  different  than  what  they  expected.  Even 
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though  there  was  not  an  exam,  the  users  associated  the  Assessment  folder  for  assignments 
rather  than  for  exams. 


Did  the  system  confuse  the  student? 

Like  easy  to  learn,  the  system  did  not  confuse  the  subjects.  In  the 
beginning  of  the  course,  they  adjusted  to  the  interface  by  clicking  on  various  buttons. 
Thereafter,  they  concentrated  on  the  material. 


Did  the  content  confuse  the  student? 

The  students  were  confused  in  the  Course  Guidelines,  Syllabus,  and 
Course  Documents  folders.  The  confusion  was  due  to  the  lack  of  organization  and  the 
already  mentioned  errors.  The  course  subject  was  pleasing,  but  the  content  needed  to  be 
structured  better. 

d.  Satisfaction 

Were  the  users  satisfied  with  the  system? 

The  users  enjoyed  the  system.  They  did  not  go  to  Help  for  assistance. 
When  they  became  misguided,  they  clicked  back  until  they  found  a  page  they  were  most 
comfortable  and  familiar  with  and  restarted. 


Were  the  users  required  to  use  “Help?” 
The  users  did  not  use  this  feature. 


Were  the  users  satisfied  with  the  course  content? 

Overall,  the  users  did  like  the  content.  Some  of  the  subjects  were  already 


familiar  with  the  information  and  skipped  to  other  topics.  They  also  enjoyed  surfing  the 
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websites  (Web  Monkey  \\as  the  most  used).  As  for  the  material  being  presented  in  small 
chunks,  most  of  the  users  felt  that  the  bits  of  information  was  sufficient.  However,  there 
should  be  a  main  textbook  for  the  course.  Furthermore,  all  subjects  noted  that  the  material 
was  for  students  who  are  interested  in  Internet  middleware  and/or  for  higher- level 
students.  Finally,  a  glossary  page  should  be  added  or  terms  should  be  listed  with  the 
assignments  so  the  student  does  not  feel  “belittled.” 
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Would  they  like  to  see  it  online  for  real  usage? 

The  subjects  did  enjoy  the  course  and  believe  that  they  could  learn  from  it 
if  it  is  placed  online  as  an  actual  course.  They  would  like  to  see  the  changes. 


2.  Analysis 

For  web-based  instruction,  system  usability  is  dependent  on  good  instructional 
design  as  instructional  design  is  dependent  on  a  good  reliable  system.  Even  though  the 
computer  system  is  running  perfectly,  the  user  may  become  dissatisfied  with  an  online 
course  because  the  content  is  not  organized  and  coherent.  If  students  spend  more  time 
wrestling  with  the  presentation  of  content  rather  than  the  content  itself,  they  begin  to 
divert  their  time  from  the  content  to  adjust  to  the  presentation.  As  a  result,  students  spend 
more  than  the  usual  amount  of  required  time  on  a  course  and  may  become  frustrated  with 
long  distance  education.  This  defeats  the  purpose  of  long  distance  education  that  allows 
students  to  take  the  course  asynchronously  for  a  short  amount  of  time. 

The  instructors  are  another  aspect  to  consider  because  they  have  to  put  the  course 
online.  If  the  instructors  do  not  have  the  proper  guidance  and  tools  to  transfer  a  traditional 
course  into  a  web-based  course,  they  also  become  frustrated  with  the  online  education 
process,  because  they  have  invested  time  into  their  courses  to  be  successful.  The 
courseware  may  function  properly,  but  if  they  do  not  know  how  to  put  the  course  online, 
then  it  is  better  to  have  no  online  course  than  a  bad  online  course. 

Instructors  also  need  to  compensate  for  a  product’s  weakness  through 
instructional  design.  Although  the  intent  of  this  research  is  not  to  judge  the  product  used, 
but  show  that  constraints  of  systems  do  affect  how  a  course  is  put  online.  Sometimes  the 
constraints  affect  the  intended  user,  which  in  this  case,  is  the  student.  Instructors  need  to 
put  on  their  creative  hats  to  compensate  for  the  constraints  by  adding  audio  and/or  text 
related  photos.  By  the  same  token,  by  placing  the  course  contents  on  file  and  having 
students  link  to  the  file  from  the  courseware  site  may  compensate  for  the  lack  of  bold, 
indents,  and  reduce  typos.  However,  the  increased  linking  can  also  hinder  learning  due  to 
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more  navigating  which  increases  the  chance  of  becoming  “lost”,  and  the  file  document 
may  not  be  presented  as  well  via  the  system. 

The  guideline  checklist  reeds  to  be  reworded  so  that  the  instructor  or  students  do 
not  just  “check  the  block.”  For  instance,  guideline  11  says,  a  syllabus  (class  schedule)  is 
in  place.  It  should  also  be  followed  by  how  clear  is  the  syllabus  and  is  it  presented  in 
small  chunks!  After  all,  the  intent  of  the  checklist  is  to  assist  instructors  to  put  an 
effective  course  online,  and  for  students  to  recognize  what  basic  ingredients  are  necessary 
to  constitute  a  good  online  course.  A  heuristics  evaluation  using  better  benchmarks  to 
measure  the  users’  behaviors. 

The  course  content  needs  to  be  revised.  More  indentation,  bold,  and  hyperlinks 
should  be  placed  in  better  strategic  places. 

Finally,  the  broad  range  of  participants  from  various  curricula  provided  a  very 
good  insight  of  how  people  view  the  course.  The  student  from  the  Information 
Technology  department  was  more  interested  in  the  buttons  whereas  the  student  from 
System  Management  cared  more  about  presentation  (fonts,  indents).  For  future 
considerations,  more  testing  involving  various  backgrounds  of  students  to  determine  how 
effective  a  course  could  be  tailored  according  to  an  improved  checklist. 
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Project  5  Evaluator’s  Task  List 

Reference  p.  50  of  Class  Notes,  Develop  the  Experiment 


Question:  How  do  we  demonstrate  that  the  Instructional  Design  Principles 
enhance  web- based  instruction?  Does  the  usability  (computer  interface) 
affect/distract  the  user  from  the  content?  How  effective  is  the  questionnaire  as  a 
measuring  tool  for  web- based  instruction? 

Note:  Due  to  time  constraints,  we  will  not  test  benchmark  four.  It  will  remain  in 
the  benchmarks  for  future  consideration. 

Benchmark  1:  Does  the  student  understand  the  instructions  on  the  Welcome 
Page?  Student  does  not  feel  “overwhelmed”  by  the  content.  (Information  is  presented  in 
small  chunks). 

Ask  the  subject  to  read  and  explain  the  content. 

Benchmark  2:  Is  the  student  comfortable  with  course  instructions?  Student 
navigates  to  Course  Information  Documents.  Student  does  not  feel  “overloaded”  by  the 
content. 

a.  Read  course  objectives 

b.  Read  syllabus  (class  schedule) 

c.  Read  course  guidelines 

d.  Explain  a  -  c. 

Benchmark  3:  Is  the  student  comfortable  with  course  instructions? 

Student  navigates  to  Course  Documents.  Student  does  not  feel  “overloaded”  by 
the  content. 

a.  Read  course  objectives 

b.  Explain  material 

Benchmark  4:  Does  the  student  comprehend  presented  material? 

Student  reads  material,  takes  a  quiz,  and  passes  quiz.  Student  does  not  feel 
“overloaded”  by  the  content. 

a.  Read  material. 

b.  Explain  material 
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Project  5  Participant’s  Task  List 
Reference  p.  50  of  Class  Notes,  Develop  the  Experiment 


You  are  testing  the  course  material.  The  expected  time  is  30  minutes.  During  this 
time,  you  may  read  and  think  aloud.  When  you  are  finished  with  the  mini- course,  we  will 
ask  questions  pertaining  to  the  material.  This  part  will  last  no  longer  than  20  minutes. 
Whenever  the  interface  distracts  you  from  the  content,  please  say  what  feature  distracts 
you  and  why. 


Instructions  for  the  course 

1.  Read  the  material  on  the  “Welcome”  Page. 

2.  Read  the  material  on  the  “Course  Information”  Page. 

3.  Read  the  material  in  the  “Course  Documents”  page. 

4.  Explain  all  sections. 

5.  Using  the  questionnaire,  rate  the  sections  and  the  material. 
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Project  5  Consent  Form 


1.  Introduction.  You  are  invited  to  participate  in  a  study  of  web-based  instruction.  With 
information  gathered  from  you  and  other  participants,  we  hope  to  discover  insight  on  how 
Instmctional  Design  principles  and  Human  Computer  Interface/Usability  principles  affect 
online  learning.  We  ask  you  to  read  and  sign  this  form  indicating  that  you  agree  to  be  in  the 
study.  Please  ask  any  questions  you  may  have  before  signing. 

2.  Background  Information.  The  Naval  Postgraduate  School  Distance  learning  is  conducting 
this  study. 

3.  Procedures.  If  you  agree  to  participate  in  this  study,  the  researcher  will  explain  the  tasks  in 
detail.  There  will  be  two  sessions:  a)  30  minute  mini-course  at  a  web  site  and  2)  a  post 
interview  discussing  the  content  of  the  course  and  any  distractions  the  system  caused  to  the 
subject. 

4.  Risks  and  Benefits.  This  research  involves  no  risks  or  discomforts.  The  benefits  to  the 
participants  are  understanding  web-based  instmction  and  contributing  to  current  research  in 
web-based  instruction. 

5.  Compensation.  No  tangible  reward  will  be  given.  A  copy  of  the  results  will  be  available  to 
you  at  the  conclusion  of  the  experiment. 

6.  Confidentiality.  The  records  of  this  study  will  be  kept  confidential.  No  information  will  be 
publicly  accessible  which  will  possibly  identify  you  as  a  participant. 

7.  Voluntary  Nature  of  the  Study.  If  you  agree  to  participate,  you  are  free  to  withdraw  from  the 
study  at  any  time  without  prejudice.  You  will  be  provided  a  copy  of  this  form  for  your  records. 

8.  Points  of  Contact.  If  you  have  any  further  questions  or  comments  after  the  completion  of  the 
study,  you  may  contact  the  research  supervisors,  Capt  Erich  I  Stefanyshyn  or  Lt  Mehmet 
Sezgin  at  (408)  656  -  4071  (Email:  eistefan@  .nps.navy.mil  or  msezgin@nps.navy.mil). 

9.  Statement  of  Consent.  I  have  read  the  above  information.  I  have  asked  all  question  and  have 
had  my  questions  answered.  I  agree  to  participate  in  this  study. 


Participant’s  Signature 

Date 

Researcher’s  Signature 

Date 

Researcher’s  Signature 

Date 

85 


Project  5  MINIMAL  RISK  CONSENT  STATEMENT 
NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 


Subj:  VOLUNTARY  CONSENT  TO  BE  A  RESEARCH  PARTICIPANT  IN:  WEB-BASED 

INSTRUCTION  EXPERIMENT 

1.  I  have  read,  understand  and  been  provided  “Information  for  Participants”  that  provides  the 
details  of  the  below  acknowledgements. 

2.  I  understand  that  this  project  involves  research.  An  explanation  of  the  purposes  of  the 
research,  a  description  of  procedures  to  be  used,  identification  of  experimental  procedures, 
and  the  extended  duration  of  my  participation  have  been  provided  to  me. 

3.  I  understand  that  this  project  does  not  involve  more  than  minimal  risk.  I  have  been  informed 
of  any  reasonably  foreseeable  risks  or  discomforts  to  me. 

4.  I  have  been  informed  of  any  benefits  to  me  or  to  others  that  may  reasonably  be  expected  from 
the  research. 

5.  I  have  signed  a  statement  describing  the  extent  to  which  confidentiality  of  records  identifying 
me  will  be  maintained. 

6.  I  have  been  informed  that  since  the  risks  are  minimal  any  injury  I  suffer  while  participating  in 
the  experiment  will  be  at  my  own  risk  and  that  I  accept  full  responsibility  for  my  own 
medical  treatment. 

7.  I  understand  that  my  participation  in  this  project  is  voluntary.  Refusal  to  participate  will 
involve  no  penalty  or  loss  of  benefits  to  which  I  am  otherwise  entitled.  I  also  understand  that 
I  may  discontinue  participation  at  any  time  without  penalty  or  loss  of  benefits  to  which  I  am 
otherwise  entitled. 

8.  I  understand  that  the  individual  to  contact  should  I  need  answers  to  pertinent  questions  about 
the  research  are  Rudy  Darken,  Ph.D.  and  Anthony  Ciavarelli,  Ph.D.,  Principal  Investigators, 
and  about  my  rights  as  a  research  subject  or  concerning  a  research  related  injury.  A  full  and 
responsive  discussion  of  the  elements  of  this  project  and  my  consent  has  taken  place. 


Signature  of  Principal  Investigator(s) 

Date 

Signature  of  Volunteer 

Date 

Signature  of  Witness 

Date 
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Project  5  PRIVACY  ACT  STATEMENT 


1.  Authority:  Naval  Post  Graduate  School,  Monterey,  CA  93940,  Naval  Instruction 

2.  Purpose:  Web-based  instruction,  instructional  design  principles  and  Human 
Computer  Interface  (HCI)/Usability  principles  will  be  studied  to  enhance 
knowledge,  and/or  improve  long-distance  education  programs  at  the  Naval 
Postgraduate  School. 

3.  Use:  Information  on  web-based  instruction  will  be  used  for  analysis  by  the 
Departments  of  the  Navy  and  Defense,  and  other  U.S.  Government  agencies, 
provided  this  use  is  compatible  with  the  purpose  for  which  the  information  was 
collected.  Use  of  the  information  may  be  granted  to  legitimate  non- government 
agencies  or  individuals  by  the  Naval  Postgraduate  School  in  accordance  with  the 
provisions  of  the  Freedom  of  Information  Act. 

4.  Disclosure/Confidentiality: 

a.  I  have  been  assured  that  my  privacy  will  be  safeguarded.  I  will  be 
assigned  a  control  or  code  number,  which  thereafter  will  be  the  only 
identifying  entry  on  any  of  the  research  records.  The  Principal 
Investigator  will  maintain  the  cross-reference  between  name  and  control 
number.  It  will  be  decoded  only  when  beneficial  to  me  or  if  some 
circumstances,  which  are  not  apparent  at  this  time,  would  make  it  clear 
that  decoding  would  enhance  the  value  of  the  research  data.  In  all  cases, 
the  provisions  of  the  Privacy  Act  Statement  will  be  honored. 

b.  I  understand  that  a  record  of  the  information  contained  in  this  Consent 
Statement  or  derived  from  the  experiment  described  herein  will  be 
retained  permanently  at  the  Naval  Postgraduate  School  or  by  higher 
authority.  I  voluntarily  agree  to  its  disclosure  to  agencies  or  individuals 
indicated  in  paragraph  3  and  I  have  been  informed  that  failure  to  agree  to 
such  disclosure  may  negate  the  purpose  for  which  the  experiment  was 
conducted. 

c.  I  also  understand  that  disclosure  of  the  requested  information,  including 
my  Social  Security  Number,  is  voluntary. 


Signature  of  Volunteer,  Print  Name,  Grade/Rank,  DOB,  SSN,  Date 


Signature  of  Witness 


Date 
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Project  5  PARTICIPANT  DEMOGRAPHICS 


Participant  ID  (assigned  by  tester): 
Circle  the  best  choice. 

Sex:  M  F 


Age:  under  18 

20-25 

25-30 

30-35 

over  30 

Education  level: 

HS  diploma 

GED 

BS/BA 

MS  PhD 

Computer  Experience: 

None 

Novice 

Intermediate  Advanced 
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APPENDIX  B:  INSTRUCTIONAL  DESIGN-USABILITY 

QUESTIONNAIRE 


PART  1:  Getting  the  Learner  Started 

This  section  provides  a  ''big  picture  view"  of  the  course.  It  assists  students  by  providing  a 
course  description,  prerequisites  for  the  course,  course  objectives  and  goals,  navigation  of 
the  web  site,  and  instructor  biography,  a  course  syllabus  (of  chapters  to  read,  homework 
assignments,  tests,  etc),  and  course  expectations  (of  the  instructor  and  student). 

Before  continuing  with  the  course,  students  should  be  able  to  understand  course 
objectives,  syllabus,  and  expectations,  and  how  to  access  the  course  lessons. 

INSTRUCTIONS 

Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 


Not  Applicable  =  NA. 

Instructional  Quality 


1.1 

Course  Objectives 

1.1.1  Course  goals  defined 

confusing 

clear 

12  34 

5  67  8  9 

NA 

1.1.2  Brief  course  description 

confusing 

clear 

12  34 

5  67  8  9 

NA 

1.1.3 

Course  prerequisites 

confusing 

clear 

12  34 

5  67  8  9 

NA 

1.1.4 

Students  given  instructions/orientation 
on  course  navigation 

unhelpful 

helpful 

1234 

5  67  8  9 

NA 

1.1.5 

Biography  of  instructor 

unhelpful 

helpful 

12  34 

5  67  8  9 

NA 

1.2  Course  Svllabus/Class  Schedule 

1.2.1  Class  schedule  (when  class  meets)  confusing  clear 

123456789  NA 

1.2.2  Class  agenda  (amount  of  reading  per  week, 
required  assignments  due,  exam  dates) 

confusing  clear 

123456789  NA 
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1.3 


Course  Expectations 


1.3.1 

Role  of  the  instructor  was  defined 

confusing 

clear 

123456789 

NA 

1.3.2 

Students’  responsibilities 

confusing 

clear 

123456789 

NA 

1.3.3 

Grading  criteria  and  procedure 

confusing 

clear 

123456789 

NA 

Web  Design 

1.4  Course  Objectives 

1.4.1  Content  from  course  objectives 
(i.e.  font,  bold,  italics,  underlines, 

misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

1.4.2  Content  provided  in  understandable 

paragraphs  inadequate  adequate 

123456789  NA 

1.4.3  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

1.5  Course  Svllabus/Class  Schedule 

1.5.1  Content  (i.e.  font,  bold,  italics, 

underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

1 .5.2  Content  provided  in  understandable 

paragraphs  inadequate  adequate 

123456789  NA 

1.5.3  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

1.6  Course  Expectations 

1.6.1  Content  (i.e.  font,  bold,  italics, 

underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 
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1.6.2  Content  provided  in  understandable  paragraphs 


1.6.3  Content  is  meaningfully  labeled 


Please  write  your  comments  about  the  Getting 


inadequate  adequate 

123456789  NA 

illogical  logical 

123456789  NA 

the  Learner  Started  section  here: 


PART  2:  Presenting  the  Subject  Material 

This  section  evaluates  the  actual  lesson/module.  This  is  where  the  actual  learning  begins. 
The  instructor  may  have  organized  the  material  into  weekly  folders,  topic  folders,  and/or 
a  hyperlink  for  the  week  for  the  student  to  view  or  download. 

In  this  area,  the  number  of  options  for  delivering  a  lesson  ranges  from  purchasing  a 
textbook  for  the  course,  additional  reading(s)  via  the  web,  problems,  exercises, 
discussions  via  chat  rooms  and/or  discussion  boards,  written  assignments  (summaries), 
and/or  extra  web-links  to  enhance  the  lesson(s). 

INSTRUCTIONS 


Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 

Not  Applicable  =  NA. 

Instructional  Quality 


2.1  Purpose  of  the  Course 


2.1.1 

The  purpose  of  each  module/section  was 
adequately  defined 

never  always 

123456789 

NA 

2.1.2 

Learning  objectives  for  each  component  of 
the  course  were  adequately  defined 

never  always 

123456789 

NA 

2.2 

2.1.3  Weekly/module  folders  are  in  logical  order 

Instructor’s  Particination 

never  always 

123456789 

NA 

2.2.1 

Students  were  encouraged  to  collaborate 

never  always 

123456789 

NA 
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2.2.2 

Instructor  gave  learning  guidance 

never  always 

123456789 

NA 

2.2.3 

Learning  guidance  was  adequate 

never  always 

123456789 

NA 

2.3.4 

The  instructor  responded  to  students’ 
questions/concerns  regarding  material 

never  always 

123456789 

NA 

2.3.5 

Instructor’s  participation  met  student 
expectations 

never  always 

123456789 

NA 

Presentation  of  Subject  Material 

2.3.1 

Instructional  material  was  presented  at 
proper  level 

difficult  easy 

123456789 

NA 

2.3.2 

Instructional  explanations  of  material 

confusing  clear 

123456789 

NA 

2.3.3 

Instructional  explanations  included 
useful  illustrative  examples  (pictures, 
graphs,  diagrams,  etc) 

never  always 

123456789 

NA 

2.3.4 

Assigned  reading  related  to  course  objectives 

never  always 

123456789 

NA 

2.3.5 

Instructional  content  relevant  to  learning 
objectives 

never  always 

123456789 

NA 

2.3.6 

Instructional  content  in  correct  sequence 

never  always 

123456789 

NA 

2.3.7 

Supplementary  web  sites  provided  to 
enhance  learning 

never  always 

123456789 

NA 

2.3.8 

Subject  material  was  highly  interactive 

never  always 

123456789 

NA 

2.3.8. 1  Interaction  of  subject  material  useful 

never  always 

123456789 

NA 

2.3.9 

Content  engaged  the  student  in  active 
learning  tasks 

never  always 

123456789 

NA 

Assignments  for  Subject  Material 

2.4.1 

Assignment  helped  achieve 
learning  objectives 

never  always 

123456789 

NA 
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2.4.2  Review  of  material  was  provided 

2.4.3  Instructional  practice  examples  were  provided 

2.4.4  Discussion  board  was  used  during  the  course 

2.4.4. 1  Discussion  board  useful 

2.4.5  Chat  rooms  were  used  during  the  course 

2.4.4. 1  Chat  rooms  useful 

2.4.4  Written  assignments  (term  papers,  one-page 
summaries,  etc)  were  included  in  each 
assignment 

2.5  Feedback  Section  for  Students 

2.5.1  Feedback  section  on  lesson  available 
for  students 

2.5.2  Feedback  section  on  lesson  useful 

Web  Design 


never  always 

123456789 

never  always 

123456789 

never  always 

123456789 

never  always 

123456789 

never  always 

123456789 

never  always 

123456789 


never  always 

123456789 


never  always 

123456789 


never  always 

123456789 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 


2.6  Subject  material  content 

(i.e.  font,  bold,  italics,  underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

2.7  Subject  material  content  provided  in 

understandable  paragraphs  inadequate  adequate 

123456789  NA 

2.8  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

Please  write  your  comments  about  the  Presentation  of  the  Subject  Material  here: 
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PART  3:  Assessing  the  Subject  Material 

This  section  rates  the  exams,  projects,  term  papers,  etc.  of  the  subject  material. 

INSTRUCTIONS 


Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 

Not  Applicable  =  NA. 

Instructional  Quality 


3.1  Exam  questions  relevant  to  learning  objectives 


3.2  Exam  questions  presented  at  proper  level 


3.3  Exam  questions  were  understandable 


3.4  Exam  had  a  reasonable  time  limit 


3.5  Exam  provided  feedback  and  review 
when  questions  answered  incorrectly 

3.6  Instructional  content  provided  adequate 
exam  preparation 

3.7  Student  critique  section  of  exam  provided 


3.8  Assessments  other  than  exams  (papers,  projects, 
assignments,  etc)  were  used  in  this  course 


3.8.1  Assessments  relevant  to  learning  objectives 


never  always 

123456789 

never  always 

123456789 

never  always 

123456789 

never  always 

123456789 


never  always 

123456789 


never  always 

123456789 

never  always 

123456789 


never  always 

123456789 


never  always 

123456789 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 


Web  Design 

3.9  Exam  content 

(i.e.  font,  bold,  italics,  underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

3.10  Exam  content  provided  in 

understandable  paragraphs  inadequate  adequate 

123456789  NA 
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3.11  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

Please  write  your  comments  about  the  Assessing  of  Subject  Material  section  here: 


PART  4:  Assessing  the  Usability  of  the  Courseware 

Courseware  is  the  software  program  that  is  used  to  deliver  the  course  (Blackboard,  Cnet, 
Courseware,  etc)  online.  If  the  courseware  does  not  function  properly  (interface,  buttons, 
link  time,  navigation,  runs  slow),  it  may  be  difficult  for  the  student  to  concentrate  on  the 
material. 

Input  from  students  on  how  the  courseware  functions,  is  vital  for  the  success  of  web- 
based  instruction  because  if  students  cannot  use  it,  then  the  courseware  hindered  the 
learning  process. 

INSTRUCTIONS 

Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 


Not  Applicable  =  NA. 


4.1 

Student  critique  provided  for  courseware 

never  always 

123456789 

NA 

4.2 

Courseware  was  easy  to  learn 

never  always 

123456789 

NA 

4.3 

Courseware  was  easy  to  use  (after  learning  it,  did  you 
struggle  with  it  for  the  remainder  of  the  course) 

never  always 

123456789 

NA 

4.4 

Course  site  was  accessible  (connectivity,  firewall) 

never  always 

123456789 

NA 

4.5 

Course  site  required  to  much  navigation  to 
view  actual  content  (clicking  until  site  reached) 

never  always 

123456789 

NA 

4.6 

Were  there  difficulties  in  navigating  from  page  to  page 

never  always 

123456789 

NA 
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4.7 

Were  there  difficulties  in  navigating  between  lessons 

never  always 

123456789 

NA 

4.8 

Were  there  difficulties  in  navigating  from  course  page 
to  web  links  and  back  to  course  page 

never  always 

123456789 

NA 

4.9 

Were  there  difficulties  in  paging  from  top  of  page 
to  bottom  of  page 

never  always 

123456789 

NA 

4.10 

Were  there  difficulties  in  paging  from  bottom  of  page 
to  top  of  page 

never  always 

123456789 

NA 

4.11 

Were  there  difficulties  in  finding  the 
“home”  page 

never  always 

123456789 

NA 

4.12 

Were  there  difficulties  in  finding  the 
desired  web  page  (location) 

never  always 

123456789 

NA 

4.13 

Speed  of  Courseware  was  adequate  (waiting  a  long 
time  to  go  to  next  site  and  for  downloads) 

never  always 

123456789 

NA 

Please  write  your  comments  about  Assessing  the  Usability  of  the  Courseware  here: 
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APPENDIX  C:  QUESTIONNAIRE  EOR  USER  INTERACTION 

SATISEACTION 


The  following  scanned  document  is  a  more  detailed  explanation  of  the  The 
Questionnaire  for  User  Interaction  Satisfaction  (QUIS)  7.0  developed  by  the  University 
of  Maryland  to  assess  the  usability  of  computers/systems  as  explained  in  Chapter  IV.  The 
explanation  begins  on  the  following  page  due  to  the  format  of  the  scan. 

The  University  of  Maryland  granted  permission  to  reprint  this  document.  For 
more  information  regarding  QUIS  7.0  can  be  obtained  by  visiting 
www.otc.umd.edu/Gatewav/Summer98/transfer.html  or  writing  the  Office  of  Technology 
Liaison  at  otl.umail.umd.edu. 
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About  the  QUIS™,  version  7.0 


The  Questionnaire  for  User  Interaction  Satisfaction  <QU1S™)  is  a  tool  developed  by  a  multi-disciplinary 
team  of  researchers  in  the  Human-Computer  Interaction  Lab  (HCIL)  at  the  University  of  Maryland  at 
College  Park.  The  QUIS™  was  designed  to  assess  users’  subjective  satisfaction  with  specific  aspects  of  the 
human-computer  interface.  The  QUIS'''”  team  successfully  addressed  the  reliability  and  validity  problems 
found  in  other  satisfaction  measures,  creating  a  measure  that  is  highly  reli^le  across  many  types  of 
interfeces. 

QUIS''”  7.0  is  the  first  update  of  the  QUIS''”  content  since  1989.  it  contains  a  demographic  questionnaire, 
a  measure  of  overall  system  satisfaction  along  six  scales,  and  hierarchically  organized  measures  of  eleven 
specific  interface  factors  (screen  factors,  terminology  and  system  feedback,  learning  factors,  system 
capabilities,  technical  manuals,  on-line  tutorials,  multimedia,  teleconferencing,  and  software  installation). 
Each  area  measures  the  users'  overall  satisfaction  with  that  facet  of  the  interface,  as  well  as  the  factors  that 
make  up  that  facet,  on  a  9-point  scale.  The  questionnaire  is  configurable  according  to  the  needs  of  each 
interface  analysis  by  including  only  the  sections  that  are  of  interest  to  the  user. 


PART  1:  System  Experience 

This  section  measures  the  user’s  experience  with  the  system  or  systems  being  tested.  Both  length  of 
experience  and  frequency  of  use  are  measured. 

PART  2:  Past  Experience 

This  section  gauges  the  user’s  experiences  with  computer  systems  in  general. 


PART  3:  Overall  User  Reactions 

Satisfaction  along  six  high  level  interface  factors  is  measured  in  this  section. 

PART  4:  Screen 

This  section  measures  satisfaction  with  a  number  of  factors  related  to  visual  displays,  including  fonts, 
highlights,  layouts  and  navigation. 

PART  5:  Terminology  and  System  Information 

This  section  measures  satisfaction  with  system  messages,  user  feedback,  and  task  related  wording  that  the 
system  generates. 

PART  6:  Learning 

This  section  measures  the  user’s  perception  of  their  ability  to  learn  complex  system  tasks. 

PART  7:  System  Capabilities 

This  section  measures  satisfaction  with  the  system's  performance  and  reliability,  both  in  error  recovery  and 
error  prevention. 

PART  8:  Technical  Manuals  &  On-line  Help 

This  section  gauges  the  system's  help  facilities  to  determine  whether  the  information  presented  is  of  the 
proper  language,  scope,  and  granularity  to  solve  the  problems  a  user  is  likely  to  have. 

PART  9:  On-line  Tutorials 

This  section  asks  users  to  rate  a  number  of  factors  that  predict  the  usefulness  of  on-line  tutorials.  Factors 
include  structure,  ease  of  navigation,  level  of  detail,  quality  of  example  tasks,  and  overall  effectiveness  of  the 
tutorial. 
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PART  10:  Multimedia 

This  section  measures  the  subjective  perception  of  images,  sound,  video,  and  color  display  within  the 
system  across  several  factors. 

PART  11:  Teleconferencing 

The  ease  or  difficulty  of  system  installation  and  configuration,  the  fidelity  of  the  communication,  and  the 
ability  to  share  a  common  experience  through  distance  communication  determines  the  success  of  a 
teleconferencing  experience.  This  section  gauges  the  user’s  subjective  satisfaction  with  these  aspects  of 
telecommunication. 

PART  12:  Software  Installation 

The  speed,  difficulty,  flexibility,  and  level  of  feedback  provided  during  the  installation  of  a  system  are 
measured  in  this  section. 

How  to  Use  the  QUIS™:  Some  Questions  and  Answers 

Can  we  omit  questions  to  save  time  or  limit  scope? 

The  hierarchical  structure  of  this  questionnaire  allows  you  to  be  selective  in  both  the  coverage  and 
granularity  of  your  inquiry.  For  this  reason,  the  questionnaire  can  be  as  long  or  as  short  as  your  situation 
requires.  There  are  large  sections  of  the  questionnaire  that  you  may  have  no  need  to  use.  Other  sections 
will  only  need  to  include  the  main  questions  without  the  sub-questions.  In  areas  that  you  are  particularly 
interested  in,  you  may  choose  to  include  the  sub-questions  also. 

in  any  case,  we  recommend  that  you  include  the  overall  measures,  as  they  provide  a  useful  and  concise 
picture  of  the  general  usability  of  your  system. 

Can  we  add  questions  that  address  our  specific  needs? 

We  generally  expect  that  any  questions  added  that  are  in  the  "QUIS™  style"  will  have  a  reasonable  amount 
of  reliability.  It  is  common  practice  to  tune  the  QUIS™  in  this  way. 

How  might  the  QUIS^  data  be  analyzed? 

Several  approaches  have  been' taken  in  the  analysis  of  QUIS™  data. 

Forming  Data  Sets: 

Once  the  QUIS™™  data  has  been  collected,  the  first  decision  is  whether  to  analyze  the  data  as  a  whole  or 
in  groups.  Obviously,  if  you  are  comparing  two  different  pieces  of  software  you  will  want  to  group  the  data 
by .  say.  Software  Package  A  and  Software  Package  B.  On  the  other  hand,  if  you  are  interested  in  differences 
between  types  of  users  of  the  same  package,  you  would  group  on  groups  of  individuals,  say.  User  Group  1 
and  User  Group  2.  When  forming  data  sets,  we  will  be  interested  in  statistics  within  each  group  as  well  as 
hypotheses  tests  between  groups.  Let’s  first  consider  the  statistics  within  groups. 

Profiles  and  Diagnostic  Tests: 

One  of  the  most  useful  analyses,  particularly  for  iterative  testing  and  design,  is  the  profile.  The  profile 
reveals  the  strengths  and  weaknesses  of  the  software  program  or  workstation  by  showing  the  deviations  of 
the  means  above  and  below  a  criterion.  A  profile  is  generated  by  calculating  the  means  and  standard 
deviations  for  each  item  in  the  QUIS™.  TTie  means  are  then  graphed  on  a  scale  from  1  to  9  as  shown  in 
the  figure  below; 

The  midpoint  of  the  rating  scale  (5)  can  be  used  as  a  criterion.  If  the  item  is  above  5,  it  is  perceived  as 
being  better  than  an  arbitrary-,  mediocre  value.  However,  that  is  generally  not  good  enough.  We  mav  also 
use  the  overall  mean  of  the  group  as  a  criterion.  Such  a  mean  is  shown  in  the  figure. 
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Figure  0-1 

It  is  useful  to  plot  a  confidence  interval  around  each  mean  in  order  to  determine  its  reliability.  The 
confidence  interval  also  indicates  whether  the  mean  of  an  item  is  significantly  above  or  below  some 
criterion.  For  example,  if  a  95%  confidence  interval  includes  ^  within  its  boundaries,  then  it  indicates  that 
the  mean  is  not  significantly  different  from  5  at  the  .05  level  of  significance. 

The  profile  can  be  used  to  identify  the  areas  in  the  application  which  are  panicuiarly  good  or  particularly 
bad.  Start  with  the  item  having  the  lowest  mean.  Identify  flaws  in  the  software  that  may  have  led  to  this 
low  mean.  Then  go  to  the  next  lowest  item  and  repeat.  Do  this  until  you  are  satisfied  that  you  have 
identified  the  major  problems.  Then  start  with  the  item  having  the  hipest  mean.  Ask  yourself  why  this 
aspect  was  rated  so  high  and  how  it  can  be  used  to  further  enhance  the  software.  Then  go  the  next  highest 
item  and  repeat.  Again,  do  this  until  you  are  satisfied  that  you  have  identified  all  of  the  strong  points  of 
the  software. 

A  more  sensitive  and  statistically  powerful  technique  for  identifying  the  strengths  and  weaknesses  is  to  use 
a  within-subject  approach.  For  each  respondent,  compute  a  mean  of  all  of  his  or  her  ratings.  Then  for  each 
of  the  respondent’s  ratings  get  the  deviation  between  that  rating  and  the  respondent’s  mean,  (dji^  =  Xj|^  - 
M.k,  where  is  the  rating  for  item  k  by  respondent  i  and  Mj  is  the  mean  for  respondent  i  across  all 
Items).  A  simple  t-test  on  these  deviations  to  see  if  they  are  significantly  different  from  zero  for  a  particular 
item  will  indicate  whether  the  item  is  perceived  as  high  or  low  relative  to  each  respondent’s  average  rating. 
In  general,  it  is  nice  to  have  a  sample  size  of  at  least  20  for  statistical  purposes.  However,  realizing  that 
many  usability  professionals  limit  their  samples  to  about  10,  it  is  suggested  that  one  avoid  statistical  tests 
and  generalizations  by  presenting  only  means  and  focusing  on  the  highest  and  lowest  ratings. 

Comparing  Groups: 

When  your  data  is  composed  of  groups,  you  make  comparisons  between  groups  at  the  overall  level  and  at 
the  level  of  individual  items.  However,  remember  that  the  more  statistical  tests  you  run,  the  greater  your 
probability  of  a  Type  1  Error  (a  spurious  result).  To  guard  against  that,  you  should  consider  only  using  the 
.01  or  .001  level  of  significance. 

For  overall  comparisons  you  may  find  the  mean  of  the  Overall  Ratings  (3. 1  to  3.6)  for  each  respondent  in 
each  group.  Then  compare  the  group  means  using  a  t-tesl.  Means  may  also  be  computed  for  sections  of 
the  QU1S™tm  or  for  all  of  the  items  on  the  QUIS'^”™  and  compared  between  groups.  Or.  of  course,  you 
may  make  comparisons  at  the  individual  item  level.  But  again,  beware  of  Type  I  Error. 
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More  Sophisticated  Analyses: 

Th€  number  of  analyses  that  one  can  play  with  is  nearly  endless,  if  you  have  enoush  data  points.  One 
should  be  cautious  of  over-analyzing  the  data.  You  will  be  bound  to  find  something  fascinating  but 
unreliable.  Nevertheless,  some  interesting  additional  analyses  can  be  used  to  investigate  the  correlational 
structure  of  the  items.  These  analyses  can  reveal  the  underlying  importance  or  relevance  of  items  to  the 
users  and  to  overall  satisfaction.  These  include;  factor  analysis,  item  analysis,  and  hierarchical  regression 
analysis. 


THIS  PAGE  LEFT  INTENTIONALLY  BLANK 
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APPENDIX  D:  SURVEY  RESULTS 


A.  QUESTIONNAIRE  USED  FOR  EVALUATION 

This  questionnaire  measured  how  web-based  courses  can  be  rated.  The  original 
survey  was  placed  on-line,  and  due  to  its  size,  it  was  unable  to  be  transferred  into  this 
document.  To  compensate  for  this,  the  questionnaire  was  restructured  to  reflect  the  Web 
version.  The  survey  begins  on  the  following  page. 
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The  Instructional  Design-Usability  Questionnaire 


My  name  is  Erich  Stefanyshyn  and  I  am  a  student  at  the  Naval  Postgraduate 
School  (NPS)  conducting  research  with  Professors  Ciavarelli  and  Darken  on: 


•  Why  usability  and  instructional  design  principles  are  not  independent 
of  each  other  when  building  an  on-line  course. 

•  Why  the  two  must  be  taken  into  consideration  when  designing  an  on¬ 
line  course  for  the  web. 

In  order  to  conduct  research,  I  need  your  input  on  how  your  online  class  was 
presented  via  the  web. 

Background:  System  usability  and  instructional  quality  are  dependent  upon  each 
other  in  web-based  instruction.  The  user  may  become  dissatisfied  with  an  online  course 
because  the  content  is  not  organized  and  coherent,  even  though  the  computer  system  is 
running  perfectly.  If  students  spend  more  time  wrestling  with  the  presentation  of  the 
subject  material  rather  than  the  content  itself,  they  begin  to  divert  their  attention  from  the 
subject  material  to  adjust  to  the  presentation  design.  As  a  result,  students  spend  more  than 
the  usual  amount  of  required  time  on  a  course  and  become  frustrated  with  long  distance 
education. 

Likewise,  the  content  and  presentation  may  be  great,  but  if  the  system/program 
functions  poorly  (low  bandwidth,  dead-end  links,  poor  user  interface  etc),  the  learning 
process  is  equally  difficult.  This  defeats  the  purpose  of  long  distance  education  that 
allows  students  to  take  the  course  asynchronously. 

Your  Help:  The  above  description  may  (or  may  not)  have  happened  to  you.  I 
would  like  to  know  what  you  considered  to  be  a  good  course  presentation  and  what  you 
consider  bad  course  presentation. 

This  is  where  I  need  you  to  rate  your  class  with  the  Web-based  Questionnaire. 
The  Web-Based  questionnaire  was  designed  to  include  both  web  design  and  instructional 
design  consideration.  The  instructional  design  relates  to  the  instructor  teaching  the 
course,  whereas  the  web  design  pertains  to  how  the  course  was  designed. 
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Your  input  will  be  evaluated  and  analyzed  to  determine  how  NPS  ean  improve  the 
quality  of  its  web-based  instruetion  courses. 

Outline  of  the  Questionnaire 

In  order  to  standardize  the  web-based  instruction  at  NPS,  the  questionnaire  is 
broken  down  into  four  parts:  Getting  the  learner  Started  section:  provides  an  introduction 
to  the  course,  course  syllabus,  objectives,  and  course  expectations.  A  Presenting  the 
Subject  Material  section:  presents  the  actual  course  lesson(s).  An  Assessing  the  Subject 
Material  section:  pertains  to  how  well  the  exams,  term  papers,  projects,  etc.  were 
prepared.  An  Assessing  the  Usability  of  the  Courseware  section:  rates  how  well  the 
program  (software)  ran. 

There  will  be  a  more  detailed  explanation  in  each  section  so  you  do  not  have  to 
come  back  to  this  page. 

Number  of  questions:  83  (78  multiple  choice  and  five  short  (optional)  essay) 

Time  to  complete  the  questionnaire :  20-30  minutes 

Thanks  for  your  valuable  time  and  support! 

DEMOGRAPHICS 

Course 

_  SS301 1  -  Space  Systems  Technology  and  Applications,  CDR  Higgins 
_  OC2022  -  Scientific  Fortran  Programming,  Arlene  Guest 
_  MN3384  -  Acquisition  Production  and  Quality  Management,  Mike  Boudreau 

Was  this  course  delivered 

a.  Completely  online  (web-based) 

b.  Hybrid  -  some  class  meetings  plus  web 

c.  VTE  (video  teleconferencing  environment)  and  on-line 
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PART  1:  Getting  the  Learner  Started 

This  section  provides  a  ''big  picture  view"'  of  the  course.  It  assists  students  by  providing  a 
course  description,  prerequisites  for  the  course,  course  objectives  and  goals,  navigation  of 
the  web  site,  and  instructor  biography,  a  course  syllabus  (of  chapters  to  read,  homework 
assignments,  tests,  etc),  and  course  expectations  (of  the  instructor  and  student). 

Before  continuing  with  the  course,  students  should  be  able  to  understand  course 
objectives,  syllabus,  and  expectations,  and  how  to  access  the  course  lessons. 

INSTRUCTIONS 

Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 

Not  Applicable  =  NA. 

Instructional  Quality 

Course  Objectives 


1. 

Course  goals  defined 

confusing  clear 

123456789 

NA 

2. 

Brief  course  description 

confusing  clear 

123456789 

NA 

3. 

Course  prerequisites 

confusing  clear 

123456789 

NA 

4. 

Students  given  instructions/orientation 
on  course  navigation 

unhelpful  helpful 

123456789 

NA 

5. 

Biography  of  instructor 

unhelpful  helpful 

123456789 

NA 

Course  Svllabus/Class  Schedule 

6. 

Class  schedule  (when  class  meets) 

confusing  clear 

123456789 

NA 

7. 

Class  agenda  (amount  of  reading  per  week, 
required  assignments  due,  exam  dates) 

confusing  clear 

123456789 

NA 

Course  Expectations 

8.  Role  of  the  instructor  was  defined  confusing  clear 

123456789  NA 
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9.  Students’  responsibilities  confusing  clear 

123456789  NA 

10.  Grading  criteria  and  procedure  confusing  clear 

123456789  NA 

Web  Design 

Course  Objectives 

1 1 .  Content  from  course  objectives 
(i.e.  font,  bold,  italics,  underlines, 

misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

12.  Content  provided  in  understandable 

paragraphs  inadequate  adequate 

123456789  NA 

13.  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

Course  Svllabus/Class  Schedule 

14  Content  (i.e.  font,  bold,  italics, 

underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

15.  Content  provided  in  understandable 

paragraphs  inadequate  adequate 

123456789  NA 

16.  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

Course  Expectations 

17.  Content  (i.e.  font,  bold,  italics, 

underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 

1 8 .  Content  provided  in  understandable 

paragraphs  inadequate  adequate 

123456789  NA 

19.  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

20.  Please  write  your  comments  about  the  Getting  the  Learner  Started  section  here: 
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PART  2:  Presenting  the  Subject  Material 

The  instructor  may  have  organized  the  material  into  weekly  folders,  topic  folders,  and/or 
a  hyperlink  for  the  week  for  the  student  to  view  or  download. 

In  this  area,  the  number  of  options  for  delivering  a  lesson  ranges  from  purchasing  a 
textbook  for  the  course,  additional  reading(s)  via  the  web,  problems,  exercises, 
discussions  via  chat  rooms  and/or  discussion  boards,  written  assignments  (summaries), 
and/or  extra  web- links  to  enhance  the  lesson(s).  This  section  evaluates  the  actual 
lesson/module.  This  is  where  the  actual  learning  begins. 


INSTRUCTIONS 

Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 

Not  Applicable  =  NA. 


Instructional  Quality 


Purpose  of  the  Course 


21. 

The  purpose  of  each  module/section  was 
adequately  defined 

never  always 

123456789 

NA 

22. 

Learning  objectives  for  each  component  of 
the  course  were  adequately  defined 

never  always 

123456789 

NA 

23. 

Weekly/module  folders  are  in  logical  order 

never  always 

123456789 

NA 

Instructor’s  Particioation 

24. 

Students  were  encouraged  to  collaborate 

never  always 

123456789 

NA 

25. 

Instructor  gave  learning  guidance 

never  always 

123456789 

NA 

26. 

Learning  guidance  was  adequate 

never  always 

123456789 

NA 

27. 

The  instructor  responded  to  students’ 
questions/concerns  regarding  material 

never  always 

123456789  NA 
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28.  Instructor’s  participation  met  expectations  never  always 

123456789  NA 


Presentation  of  Subject  Material 


29. 

Grading  criteria  and  procedure 

difficult  easy 

123456789 

NA 

30. 

Instructional  material  was  presented  at 
proper  level 

difficult  easy 

123456789 

NA 

31. 

Instructional  explanations  of  materials 

confusing  clear 

123456789 

NA 

32. 

Instructional  explanations  included 
useful  illustrative  ejamples  (pictures, 
graphs,  diagrams,  etc) 

never  always 

123456789 

NA 

33. 

Assigned  reading  related  to  course  objectives 

never  always 

123456789 

NA 

34. 

Instructional  content  relevant  to  learning 
objectives 

never  always 

123456789 

NA 

35. 

Instructional  content  in  correct  sequence 

never  always 

123456789 

NA 

36. 

Supplementary  web  sites  provided  to 
enhance  learning 

never  always 

123456789 

NA 

37. 

Subject  material  was  highly  interactive 

never  always 

123456789 

NA 

38. 

Interaction  of  subject  material  useful 

never  always 

123456789 

NA 

39. 

Interaction  of  subject  material  useful 

never  always 

123456789 

NA 

40. 

Content  engaged  the  student  in  active 
learning  tasks 

never  always 

123456789 

NA 

Assienments  for  Subject  Material 

41. 

Assignment  helped  achieve 
learning  objectives 

never  always 

123456789 

NA 

42. 

Review  of  material  was  provided 

never  always 

123456789 

NA 

109 


43. 

Instructional  practice  examples  were  provided 

never  always 

123456789 

NA 

44. 

Discussion  board  was  used  during  the  course 

never  always 

123456789 

NA 

45. 

Discussion  board  useful 

never  always 

123456789 

NA 

46. 

Chat  rooms  were  used  during  the  course 

never  always 

123456789 

NA 

47. 

Chat  rooms  useful 

never  always 

123456789 

NA 

48. 

Written  assignments  (term  papers,  one-page 
Summaries,  etc)  were  included  in  each 
assignment 

never  always 

123456789 

NA 

Feedback  Section  for  Students 

49. 

Feedback  section  on  lesson  available 
for  students 

never  always 

123456789 

NA 

50. 

Feedback  section  on  lesson  useful 

never  always 

123456789 

NA 

Web  Design 

51. 

Subject  material  content 

(i.e.  font,  bold,  italics,  underlines, 

misspellings,  etc) 

hard  to  read  easy  to  read 

123456789 

NA 

52. 

Subject  material  content  provided  in 
understandable  paragraphs 

inadequate  adequate 

123456789 

NA 

53. 

Content  is  meaningfully  labeled 

illogical  logical 

123456789 

NA 

54. 

Please  write  your  comments  about  the  Presentation  of  Subject  Material  here: 

no 


PART  3:  Assessing  the  Subject  Material 

This  section  rates  the  exams,  projects,  term  papers,  etc.  of  the  subject  material. 
INSTRUCTIONS 

Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 


Not  Applicable  =  NA. 

Instructional  Quality 


55. 

Exam  questions  relevant  to  learning  objectives 

never  always 

123456789 

NA 

56. 

Exam  questions  was  presented  at  proper  level 

never  always 

123456789 

NA 

57. 

Exam  questions  are  understandable 

never  always 

123456789 

NA 

58. 

Exam  has  a  reasonable  time  limit 

never  always 

123456789 

NA 

59. 

Exam  provided  feedback  and  review 
when  questions  answered  incorrectly 

never  always 

123456789 

NA 

60. 

Instructional  content  provided  adequate 
exam  preparation 

never  always 

123456789 

NA 

61. 

Student  critique  section  of  exam  provided 

never  always 

123456789 

NA 

62. 

Assessments  other  than  exams  (papers,  projects, 
assignments,  etc)  were  used  in  this  course) 

never  always 

123456789 

NA 

63. 

Assessments  relevant  to  learning  objectives 

never  always 

123456789 

NA 

Web  Design 

64. 

Exam  content 

(i.e.  font,  bold,  italics,  underlines,  misspellings,  etc)  hard  to  read  easy  to  read 

123456789  NA 
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65. 


Exam  content  provided  in 

understandable  paragraphs  inadequate  adequate 

123456789  NA 


66.  Content  is  meaningfully  labeled  illogical  logical 

123456789  NA 

67.  Please  write  your  comments  about  the  Assessing  of  Subject  Material  section  here: 


PART  4:  Assessing  the  Usability  of  the  Courseware 

Courseware  is  the  software  program  that  is  used  to  deliver  the  course  (Blackboard,  Cnet, 
Courseware,  etc)  online.  If  the  courseware  does  not  function  properly  (interface,  buttons, 
link  time,  navigation,  runs  slow),  it  may  be  difficult  for  the  student  to  concentrate  on  the 
material. 

Input  from  students  on  how  the  courseware  functions,  is  vital  for  the  success  of  web- 
based  instruction  because  if  students  cannot  use  it,  then  the  courseware  hindered  the 
learning  process. 

INSTRUCTIONS 

Please  circle  the  numbers,  which  most  appropriately  reflect  your  impressions  about  this 
course. 

Not  Applicable  =  NA. 


68. 

Student  critique  provided  for  courseware 

never 

always 

1  2  345 

67  8  9 

69. 

Courseware  was  easy  to  learn 

never 

always 

1  2  345 

67  8  9 

70. 

Courseware  was  easy  to  use  (after  learning  it,  did  you 
struggle  with  it  for  the  remainder  of  the  course) 

never 

always 

1  2  345 

67  8  9 

71. 

Course  site  was  accessible  (connectivity,  firewall) 

never 

always 

1  2  345 

67  8  9 

72. 

Course  site  required  to  much  navigation  to 
view  actual  content  (clicking  until  site  reached) 

never 

always 

1  2  345 

67  8  9 
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73. 

Were  there  difficulties  in  navigating  from  page  to  page 

never  always 

123456789 

NA 

74. 

Were  there  difficulties  in  navigating  between  lessons 

never  always 

123456789 

NA 

75. 

Were  there  difficulties  in  navigating  from  course  page 
to  web  links  and  back  to  course  page 

never  always 

123456789 

NA 

76. 

Were  there  difficulties  in  paging  from  top  of  page 
to  bottom  of  page 

never  always 

123456789 

NA 

77. 

Were  there  difficulties  in  paging  from  bottom  of  page 
to  top  of  page 

never  always 

123456789 

NA 

78. 

Were  there  difficulties  in  finding  the 
“home”  page 

never  always 

123456789 

NA 

79. 

Were  there  difficulties  in  finding  the 
desired  web  page  (location) 

never  always 

123456789 

NA 

80. 

Speed  of  Courseware  was  adequate  (waiting  a  long 
time  to  go  to  next  site  and  for  downloads) 

never  always 

123456789 

NA 

81.  Please  write  your  comments  about  Assessing  the  Usability  of  the  Courseware  here: 


END  OF  QUESTIONNAIRE  !! 
Thank  you  for  your  cooperation!! 
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B.  GRAPHING  OF  RESULTS 

The  results  from  the  survey  were  interpreted  measured  on  Excel  spreadsheet  and 
then  transferred  to  the  program  Stat  View  for  graphing.  The  purpose  of  the  graphs  is  to 
compare  the  average  of  the  respondents  based  on  the  class  conditions.  The  two  conditions 
are: 

•  Hybrid  (class/web-enhanced)  and 

•  VTE  (video  teleconferencing/web- enhanced). 

They  offer  an  opportunity  to  compare  not  only  two  different  groups,  but  also  the 
chance  of  evaluating  how  well  the  questionnaire  can  measure  web-based  instruction.  The 
results  show  that  the  questionnaire  does  deliver  constant  results  and  can  measure  web- 
based  instruction. 

The  averages  were  obtained  by  the  rating  system  provided  by  the  QUIS  7.0 
(APPENDIX  C:  QUESTIONNAIRE  EOR  USER  INTERACTION  SATISEACTION). 
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Graph  2:  Getting  the  Learner  Started 
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3.  Graph  3:  Presenting  the  Subject  Material 
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4.  Graph  4:  Assessing  the  Subject  Material 


Cell  Line  Chart 
Split  By:  Condition 
Row  exclusion:  Erich2.xls  (imported) 

9 

8 
7 

c  6 
cc 

^  5 

o  4 
O 

3 
2 
1 
0 


LO 

CD 

1^ 

00 

CD 

o 

CM 

CO 

■M- 

LO 

CD 

LO 

LO 

CD 

un 

LO 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

o 

o 

o 

O 

O 

o 

o 

O 

O 

O 

O 

O 

118 


5.  Graph  5:  Assessing  the  Usability  of  the  Courseware 
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